Image fusion method, device and electronic equipment

Abstract

The present application provides an image fusion method, device and electronic equipment. In the embodiment, the first image data and the second image data to be fused are fused by the trained first fusion weight matrix and the second fusion weight matrix to obtain a fused image. When applied to a security check scene, a fused security check image can be finally obtained. Based on the fused security check image, the training of a contraband detection model and the like is performed for contraband detection, and the efficiency and accuracy of contraband detection are improved.

Description

Technical Field

[0001] This application relates to image processing technology, and more particularly to image fusion methods, apparatus and electronic devices. Background Technology

[0002] In many application scenarios, such as airports, train stations, subway entrances, and important event venues, security checks, such as using X-ray security scanners to inspect luggage, are often conducted to ensure safety and reduce risks.

[0003] In current security inspection applications, security personnel manually review images scanned by security inspection machines to determine whether there are any non-safety items (also known as prohibited items) that could affect security. However, this manual review method relies too heavily on security personnel and poses many security risks. Summary of the Invention

[0004] This application provides an image fusion method, apparatus, and electronic device for intelligent security inspection based on fused images.

[0005] The technical solution provided in this application includes:

[0006] An image fusion method, applied to an electronic device, includes:

[0007] Obtain the first and second image data to be fused;

[0008] The first image data and the second image data are input into the trained target fusion network to fuse at least the first result and the second result to obtain fused image data. The first result refers to the result obtained by performing a specified operation on the feature value of each pixel in the first image data and the weight corresponding to that pixel in the first fusion weight matrix. The second result refers to the result obtained by performing a specified operation on the feature value of each pixel in the second image data and the weight corresponding to that pixel in the second fusion weight matrix. The first fusion weight matrix and the second fusion weight matrix are the model parameters trained by the target fusion network.

[0009] An image fusion apparatus, applied to an electronic device, comprising:

[0010] The acquisition unit is used to acquire the first image data and the second image data to be fused.

[0011] A fusion unit is used to input the first image data and the second image data into a trained target fusion network, so as to fuse at least the first result and the second result through the target fusion network to obtain a fused image; the first result refers to the result obtained by performing a specified operation on the feature value of each pixel in the first image data and the weight corresponding to that pixel in the first fusion weight matrix; the second result refers to the result obtained by performing a specified operation on the feature value of each pixel in the second image data and the weight corresponding to that pixel in the second fusion weight matrix; the first fusion weight matrix and the second fusion weight matrix are model parameters trained by the target fusion network.

[0012] An electronic device comprising: a processor and a machine-readable storage medium;

[0013] The machine-readable storage medium stores machine-executable instructions that can be executed by the processor;

[0014] The processor is used to execute machine-executable instructions to implement the steps in the method described above.

[0015] As can be seen from the above technical solutions, in this embodiment, the first image data and the second image data to be fused are fused using the trained first fusion weight matrix and the second fusion weight matrix to obtain a fused image. When applied to a security inspection scenario, a fused security inspection image can be obtained. Based on this fused security inspection image, a model for detecting prohibited items can be trained to perform other tasks such as detecting prohibited items, thereby improving the efficiency and accuracy of prohibited item detection.

[0016] Furthermore, in this embodiment, by randomly fusing existing labeled image data, the diversity of image data can be enriched without increasing the cost of any data labeling. This can be applied to security inspection scenarios, which can solve the problems of high cost and unreliable labeling quality of X-ray false color images in security inspections.

[0017] Furthermore, this image fusion method in this embodiment can jointly train data with different annotation specifications, indirectly improving the generalization performance of the contraband detection model. Attached Figure Description

[0018] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0019] Figure 1 A flowchart illustrating the method provided in this application embodiment;

[0020] Figure 2 A fusion network structure diagram provided for embodiments of this application;

[0021] Figure 3 A flowchart illustrating the implementation of step 102 provided in this application embodiment;

[0022] Figure 4 This is a diagram of the self-encoder network structure provided in an embodiment of this application;

[0023] Figure 5 This is a diagram of the autoencoder network training framework provided in the embodiments of this application;

[0024] Figure 6 This is a diagram illustrating the training framework of the fusion network provided in an embodiment of this application.

[0025] Figure 7 A fusion network structure diagram provided for embodiments of this application;

[0026] Figure 8 This is a diagram of the discriminator network structure provided in the embodiments of this application;

[0027] Figure 9 This is a structural diagram of the device provided in the embodiments of this application;

[0028] Figure 10 This is a hardware structure diagram of the device provided in the embodiments of this application. Detailed Implementation

[0029] With the continuous development of deep learning, high-accuracy and high-speed contraband detection models are commonly used for automatic contraband detection. However, training these models requires massive amounts of labeled training data (also known as security contraband data), which is scarce in practical applications. Furthermore, security checks often encounter situations where multiple items are stacked together (i.e., obscuring each other), affecting the accuracy of contraband detection and making the data even more difficult to collect and label. This further increases the difficulty of using contraband detection models.

[0030] To address the aforementioned problems, this embodiment provides an automated deep learning-based image fusion method for simulating security inspection contraband data in complex scenarios such as multiple items stacked or occluded. This simulated data is used to train a contraband detection model for contraband detection, thereby improving the generalization performance of the contraband detection system. To make the method provided in this application easier to understand, the method is described in detail below with reference to the accompanying drawings and embodiments:

[0031] See Figure 1 , Figure 1This is a flowchart illustrating a method provided in an embodiment of this application. This process can be applied to electronic devices. In one example, the electronic device may be an X-ray-based security scanner or analyzer, etc., but this embodiment is not specifically limited to this.

[0032] like Figure 1 As shown, the process may include the following steps:

[0033] Step 101: Obtain the first image data and the second image data to be fused.

[0034] As one embodiment, the first image data and the second image data are respectively a first image and a second image acquired via an acquisition device.

[0035] In another embodiment, the first image data is a first image captured by the acquisition device, which is then input into the encoder of the trained autoencoder network, and the encoder extracts a first feature image, such as semantic feature data. The second image data is a second image captured by the acquisition device, which is then input into the encoder, and the encoder extracts a second feature image, such as semantic feature data. The autoencoder network will be described below, and will not be repeated here.

[0036] Step 102: Input the first image data and the second image data into the trained target fusion network to fuse at least the first result and the second result to obtain fused image data. The first result refers to the result obtained by performing a specified operation on the feature value of each pixel in the first image data and the weight corresponding to that pixel in the first fusion weight matrix. The second result refers to the result obtained by performing a specified operation on the feature value of each pixel in the second image data and the weight corresponding to that pixel in the second fusion weight matrix. The first fusion weight matrix and the second fusion weight matrix are the model parameters trained by the target fusion network.

[0037] Compared to the generalized weights configured in traditional weighted fusion methods, this embodiment sets specific fusion weights for each pixel in the image data. For example, the first fusion weight matrix is ​​composed of the weights corresponding to each pixel in the first image data, and the second fusion weight matrix is ​​composed of the weights corresponding to each pixel in the second image data. This can improve the fusion effect and achieve true fusion of the first and second image data.

[0038] Furthermore, in this embodiment, the first and second fusion weight matrices are not set based on experience, but are obtained by training the target fusion network using the training images. They are represented as the model parameters of the trained target fusion network. Additionally, this embodiment considers the "penetration" characteristic of the image when training the target fusion network. This ensures that the fused image produced by the finally trained target fusion network possesses the "penetration" characteristic of a security X-ray pseudo-color image. The training method for the target fusion network will be described with examples below, and will not be elaborated upon here.

[0039] In applications, such as security checks, the variety of contraband in X-ray pseudo-color images is significant due to the diverse types of contraband, varying material textures, and frequent stacking. This results in high annotation costs and inconsistent annotation quality. However, this embodiment, based on steps 101 and 102, allows for the random fusion of existing labeled X-ray pseudo-color images. The fused image data includes the annotation information from each image before fusion, enriching data diversity without increasing annotation costs. Furthermore, in security checks, the varying hazard levels of contraband lead to a scarcity of high-risk data. Therefore, labeling often prioritizes high-risk contraband, potentially resulting in different annotation standards across different security datasets, preventing direct integration for training. This embodiment, however, allows for joint training of datasets with varying annotation standards through fusion, indirectly improving the generalization performance of the contraband detection model.

[0040] This concludes the process. Figure 1 The process is shown below.

[0041] pass Figure 1 As can be seen from the process shown, in this embodiment, the first image data and the second image data to be fused are fused using the trained first fusion weight matrix and the second fusion weight matrix to obtain a fused image. When applied to a security inspection scenario, a fused security inspection image can be obtained. Based on this fused security inspection image, a model for detecting prohibited items can be trained to perform other tasks such as detecting prohibited items, thereby improving the efficiency and accuracy of prohibited item detection.

[0042] Furthermore, in this embodiment, by randomly fusing existing labeled image data, the diversity of image data can be enriched without increasing the cost of any data labeling. This can be applied to security inspection scenarios, which can solve the problems of high cost and unreliable labeling quality of X-ray false color images in security inspections.

[0043] Furthermore, this image fusion method in this embodiment can jointly train data with different annotation specifications, indirectly improving the generalization performance of the contraband detection model.

[0044] The following describes how the target fusion network fuses at least the first and second results in step 102:

[0045] As an example, the network architecture of the target fusion network in this embodiment is as follows: Figure 2 As shown. Based on Figure 2 The network architecture of the target fusion network shown is as follows: Figure 3 An example illustrates how, in step 102, the target fusion network fuses at least the first and second results:

[0046] See Figure 3 , Figure 3 A flowchart illustrating the implementation of step 102 in an embodiment of this application. Figure 3 As shown, the process may include the following steps:

[0047] Step 301: The feature values ​​of pixels at the same position in the first result and the second result are fused to obtain the third result; the first semantic feature is extracted from the third result by the first feature extractor in the target fusion network.

[0048] In this embodiment, the first image data and the second image data have the same size, and similarly, the first result and the second result also have the same size. Under this premise, the feature values ​​of pixels at the same positions in the first result and the second result can be fused, such as by addition, to obtain the third result.

[0049] Subsequently, as an example, the first semantic features can also be extracted from the third result by the first feature extractor in the target fusion network. Here, the first feature extractor is generally composed of convolutional layers, normalization layers, etc., and this example is not specifically limited.

[0050] Step 302: Perform channel-level stitching operation on the first image data and the second image data to obtain stitched image data, and extract the second semantic features from the stitched image data through the second feature extractor in the target fusion network.

[0051] Here, the structure of the second feature extractor is similar to that of the first feature extractor described above, and it is generally composed of convolutional layers, normalization layers, etc. This embodiment is not specifically limited.

[0052] Step 303: The first semantic feature and the second semantic feature are fused to obtain fused image data.

[0053] Optionally, in this embodiment, the first semantic feature and the second semantic feature can be added pixel by pixel. For example, the semantic feature values ​​at the same position in the first semantic feature and the second semantic feature can be added to obtain fused image data.

[0054] This concludes the process. Figure 3 The process is shown below.

[0055] pass Figure 3 The process shown demonstrates how to fuse at least the first and second results using a target fusion network to obtain a fused image.

[0056] The autoencoder network described above is described below:

[0057] In this embodiment, the structure of the autoencoder network is as follows: Figure 4 As shown, it includes an encoder and a decoder. Here, the encoder is used to encode input data, such as images or features, into a set of vectors. The decoder is used to restore the input data, such as the input image or input features mentioned above, from the output of the encoder.

[0058] It should be noted that, in Figure 4 The feature extractors shown, whether in the encoder or decoder, are generally obtained by assembling convolutional layers in a cascaded or parallel manner. Figure 4 In this process, downsampling is generally achieved through pooling structures or interpolation; upsampling is generally achieved through deconvolution layers or interpolation; and channel mapping is generally achieved through convolutional layers.

[0059] In this embodiment, the autoencoder network training can be performed as follows: Figure 5 The framework diagram shown. Figure 5 In the context of training, the autoencoder network uses the loss function L. auto This involves pixel-level loss (P-Loss) and feature-level loss (F-Loss).

[0060] Optionally, the above loss function L auto It can be expressed by the following formula: L auto =αL pixel ×βL feat ;

[0061] As an example, L pixel Indicate I img and O img The pixel-level loss P-Loss between I. img O represents the training images used to train the autoencoder network. img Indicates that I feat The output I obtained by inputting into the decoder in the autoencoder network to be trained. featThis refers to I img The feature image is input to the encoder in the autoencoder network to be trained. As an example, L... pixel L is expressed by the following formula: pixel =MAE(I img O img MAE stands for Mean Absolute Error.

[0062] As an example, L feat Indicate I feat With O feat Feature-level loss F-Loss, I feat As described above, O feat O img The feature image obtained after inputting into the third-party encoder. Here, the third-party encoder in the dashed box is newly added during the training of the autoencoder network. It is used to obtain the output of the decoder and encode the output to calculate the F-Loss. As an example, L feat L is expressed by the following formula: feat =MSE(I feat O feat MSE stands for Mean Square Error.

[0063] As an example, α and β are preset values, which can be set according to actual needs, and this example is not specifically limited.

[0064] After training the autoencoder network described above, the encoder and decoder in the autoencoder network can be used independently. The encoder is used to encode input information such as input images or input features, while the decoder is used to restore the output of the encoder to the input information such as input images or input features.

[0065] The target fusion network described above is described below:

[0066] In this embodiment, the target fusion network is trained through the following steps: adversarial network training is performed on the fusion network to be trained and the already deployed discriminator network; the fusion network to be trained is deployed between the encoder and decoder of the already trained autoencoder network, and the discriminator network is deployed after the decoder; during adversarial network training, if the loss function L of the fusion network to be trained... g Satisfying the corresponding fusion network loss requirements and the discriminator network loss function L d If the corresponding discriminator network loss requirement is met, the current fusion network to be trained can be used as the target fusion network mentioned above.

[0067] In this embodiment, as described above, the fusion network is embedded between the encoder and decoder in the autoencoder network for training. This ensures that the fusion network is trained based on the semantic features extracted by the encoder, which helps the fusion network learn higher-level abstract features and more general model parameters, thereby improving the generalization performance of the fusion network. Figure 6 An example is shown in the training framework structure diagram of the fusion network to be trained (the target fusion network is similar).

[0068] In the above description, the loss function L of the fusion network to be trained is... g This can be expressed by the following formula:

[0069] L g =θ1×L f +θ2×BCE(D(O), 1).

[0070] Wherein, θ1 and θ2 are preset values, which can be set according to actual needs, and this embodiment is not specifically limited. In the above description, L f Let represent a loss function that reflects the X-ray transparency characteristics, as illustrated below. D(O) represents the output of the discriminator network after inputting the fused image O. The fused image O refers to the output of the decoder after inputting the fused feature image O′. The fused feature image O′ refers to the output of the fusion of the training feature images I1′ and I2′ into the current fusion network to be trained. The training feature image I1′ refers to the output of the encoder after inputting the training image I1, and the training feature image I2′ refers to the output of the encoder after inputting the training image I2. BCE() represents the binary cross-entropy loss used for binary classification.

[0071] In this embodiment, the network to be trained in the fusion network is as follows: Figure 7 As shown, IN1 and IN2 represent the inputs to the fusion network, such as the training feature images I1′ and I2′ mentioned above, respectively. OUT represents the output of the fusion network, such as the fused feature image O′ mentioned above. This indicates a pixel-by-pixel addition operation (such as adding the feature values ​​of pixels at the same position in two different inputs). This indicates a pixel-wise multiplication operation (such as multiplying the feature values ​​of pixels at the same position in two different inputs). This indicates a channel-level splicing operation.

[0072] As an example, Figure 7The pixel-level weight extractor is used to extract the fusion weights corresponding to each pixel in the input data, such as the training feature image I1′ mentioned above. It is generally composed of convolutional layers, activation functions, etc. The feature extractor is used to extract semantic features from the input data. It is generally composed of convolutional layers, normalization layers, etc.

[0073] In security inspection scenarios, the most prominent feature of security X-ray images is "transparency," meaning that even after multiple objects are stacked and imaged, the outline and corresponding color of each object can still be clearly seen. Based on this, this embodiment also considers that the image fused by the final trained target fusion network should also possess the "transparency" characteristic unique to security inspection images.

[0074] As an example, this can be achieved through the L described above. f Characterizes X-ray transillumination properties. Optionally, L f L is expressed by the following formula: f =abs(0.5,L) f1 )+abs(0.5,L f2 )+abs(L f1 L f2 ).

[0075] Among them, L f1 L represents the similarity between the training feature image I1′ and the fused feature image O′. f1 =SSIM(I1′,O′)L f2 L represents the similarity between the training feature image I2′ and the fused feature image O′. f2 =SSIM(I2′,O′); SSIM stands for Structural Similarity Index. Specifically, SSIM is a number between 0 and 1. The larger the SSIM, the smaller the difference between the output and input of the fusion network, that is, the better the image quality.

[0076] Optionally, when the similarity between the training feature image I1′ and the fused feature image O′ is a first preset similarity, such as 0.5, and the similarity between the training feature image I2′ and the fused feature image O′ is a second preset similarity, such as 0.5, the transmission characteristics of the fusion network are optimal. In other words, this embodiment can control the similarity between the training feature image I1′ and the fused feature image O′ to a first preset similarity, such as 0.5, and the similarity between the training feature image I2′ and the fused feature image O′ to a second preset similarity, such as 0.5, during the training of the fusion network.

[0077] In addition, in order to integrate the problems left over from the internet, namely the problem of unreality, such as Figure 6As shown, a discriminator network is connected after the decoder. This discriminator network undergoes adversarial training with the fusion network, aiming to generate images that the discriminator cannot recognize. It should be noted that, theoretically, training the fusion network requires supervised training with actual fused images. However, in practice, it is difficult or even impossible to obtain fused images. Therefore, this embodiment integrates the discriminator network after the fusion network to determine the authenticity of its output fused image. This way, during training, actual fused images are no longer needed; only the training set for the discriminator network needs to be real security inspection images.

[0078] In this embodiment, the loss function L of the discriminator network d This can be expressed by the following formula:

[0079] L d =δ1×BCE(D(I1),1)+δ2×BCE(D(I2),1))+δ3×BCE(D(O),0);

[0080] Wherein, δ1, δ2, and δ3 are preset values, which can be set according to actual needs and are not limited here. D(I1) represents the output result obtained after inputting the training image I1 into the discriminator network; D(I2) represents the output result obtained after inputting the training image I2 into the discriminator network. D(O) is as described above and will not be repeated here.

[0081] As an example, in a specific implementation, the discriminator network can be a spectral normalized discriminator. By implementing the discriminator network using a spectral normalized discriminator, the spectral norm of the weight matrix of each layer can be constrained, resulting in better insensitivity to input perturbations and thus enhancing stability during adversarial training. Figure 8 An example is shown illustrating the structure of a spectral normalization discriminator.

[0082] pass Figure 8 As shown in the diagram, the spectral normalization discriminator, in addition to including a spectral normalization layer, further includes shallow feature extractors, mid-level feature extractors, and deep feature extractors. In a specific implementation, the spectral normalization discriminator may contain one shallow feature extractor, four mid-level feature extractors, and two deep feature extractors.

[0083] The shallow feature extractor is used to extract features from the input, mainly extracting low-level semantic features; the medium-level feature extractor extracts features based on the output of the shallow feature extractor, mainly extracting high-level semantic features; and the deep feature extractor performs further feature extraction based on the output of the medium-level feature extractor module, mainly extracting more abstract high-level semantic features.

[0084] In this embodiment, the shallow, medium, and deep feature extractors have similar structures, generally consisting of convolutional layers, pooling layers, and activation layers. The shallow and medium feature extractors are quite similar; after each pass through a shallow or medium feature extractor, the output size is reduced to half the original size compared to the input. However, the deep feature extractor does not exhibit any size change.

[0085] The training of the fusion network provided in this embodiment has been described above with examples. The apparatus provided in this application is described below:

[0086] See Figure 9 , Figure 9 A structural diagram of a device provided in an embodiment of this application. This device is applied to the aforementioned electronic device and includes:

[0087] The acquisition unit is used to acquire the first image data and the second image data to be fused.

[0088] A fusion unit is used to input the first image data and the second image data into a trained target fusion network, so as to fuse at least the first result and the second result through the target fusion network to obtain a fused image; the first result refers to the result obtained by performing a specified operation on the feature value of each pixel in the first image data and the weight corresponding to that pixel in the first fusion weight matrix; the second result refers to the result obtained by performing a specified operation on the feature value of each pixel in the second image data and the weight corresponding to that pixel in the second fusion weight matrix; the first fusion weight matrix and the second fusion weight matrix are model parameters trained by the target fusion network.

[0089] Optionally, the first image data and the second image data are respectively a first image and a second image acquired via an acquisition device; or,

[0090] The target fusion network is embedded between the encoder and decoder of the trained autoencoder network;

[0091] The first image data is the first feature image extracted by the encoder after the first image acquired by the acquisition device is input to the encoder; the second image data is the second feature image extracted by the encoder after the second image acquired by the acquisition device is input to the encoder.

[0092] Optionally, the step of fusing at least the first result and the second result through the target fusion network to obtain the fused image includes:

[0093] The feature values ​​of pixels at the same position in the first and second results are fused to obtain the third result; the first semantic feature is extracted from the third result by the first feature extractor in the target fusion network;

[0094] The first image data and the second image data are spliced ​​at the channel level to obtain spliced ​​image data. The second semantic feature is extracted from the spliced ​​image data by the second feature extractor in the target fusion network.

[0095] The first semantic feature and the second semantic feature are fused to obtain the fused image.

[0096] Optionally, the above-mentioned target fusion network is trained through the following steps:

[0097] The fusion network to be trained and the already deployed discriminator network are trained as an adversarial network; the fusion network to be trained is deployed between the encoder and decoder of the already trained autoencoder network, and the discriminator network is deployed after the decoder.

[0098] When training an adversarial network, if the loss function L of the fusion network to be trained is... g The loss function L of the discriminator network satisfies the corresponding fusion network loss requirements. d If the corresponding discriminator network loss requirement is met, the current fusion network to be trained can be used as the target fusion network.

[0099] Optionally, the loss function L g This can be expressed by the following formula:

[0100] L g =θ1×L f +θ2×BCE(D(O), 1);

[0101] Where θ1 and θ2 are preset values, L f Let D(O) represent a loss function that reflects the X-ray transparency characteristics. D(O) represents the output result obtained after inputting the fused image O into the discriminator network. The fused image O refers to the output result obtained by inputting the fused feature image O′ into the decoder. The fused feature image O′ refers to the output result obtained by inputting the training feature images I1′ and I2′ into the fusion network to be trained. The training feature image I1′ refers to the output result obtained by inputting the training image I1 into the encoder. The training feature image I2′ refers to the output result obtained by inputting the training image I2 into the encoder. BCE() represents the binary cross-entropy loss used for binary classification.

[0102] Optionally, the loss function L of the discriminator network dThis can be expressed by the following formula:

[0103] L d =δ1×BCE(D(I1),1)+δ2×BCE(D(I2),1))+δ3×BCE(D(O),0);

[0104] Wherein, δ1, δ2, and δ3 are preset values, which can be set according to actual needs and are not limited here. D(I1) represents the output result obtained after inputting the training image I1 into the discriminator network; D(I2) represents the output result obtained after inputting the training image I2 into the discriminator network. D(O) is as described above and will not be repeated here.

[0105] Optionally, the L f This can be expressed by the following formula:

[0106] L f =abs(0.5,L) f1 )+abs(0.5,L f2 )+abs(L f1 L f2 );

[0107] Among them, L f1 L represents the similarity between the training feature image I1′ and the fused feature image O′. f1 =SSIM(I1′,O′)L f2 L represents the similarity between the training feature image I2′ and the fused feature image O′. f2 =SSIM(I2′, O′); SSIM represents the structural similarity index;

[0108] When the similarity between the training feature image I1′ and the fused feature image O′ is a first set similarity, and the similarity between the training feature image I2′ and the fused feature image O′ is a second set similarity, the transmission characteristics of the fusion network are optimal.

[0109] Optionally, the discriminator network is a spectral normalized discriminator network.

[0110] Optionally, the loss function L used during training of the autoencoder network... auto L is expressed by the following formula: auto =αL pixel ×βL feat ;

[0111] Among them, L pixel Indicate I img and O img Pixel-level loss between, I img O represents the training images used to train the autoencoder network. img Indicates that Ifeat The output I obtained by inputting into the decoder in the autoencoder network to be trained. feat This refers to I img The feature image obtained by the encoder in the autoencoder network to be trained; L pixel L is expressed by the following formula: pixel =MAE(I img O img MAE represents the mean absolute error;

[0112] L feat Indicate I feat With O feat Feature-level loss between, O feat O img The feature image obtained after inputting to a third-party encoder, whereby the third-party encoder is a newly added encoder during the training of the autoencoder network to assist in calculating the feature-level loss, L feat L is expressed by the following formula: feat =MSE(I feat O feat MSE represents the mean squared error loss; α and β are preset values.

[0113] This concludes the process. Figure 9 Structural description of the device shown.

[0114] Correspondingly, this application also provides Figure 9 The hardware structure of the device shown. See also Figure 10 The hardware structure may include: a processor and a machine-readable storage medium, the machine-readable storage medium storing machine-executable instructions that can be executed by the processor; the processor is used to execute the machine-executable instructions to implement the method disclosed in the above example of this application.

[0115] Based on the same application concept as the above method, this application embodiment also provides a machine-readable storage medium storing a plurality of computer instructions, which, when executed by a processor, can implement the method disclosed in the above examples of this application.

[0116] For example, the aforementioned machine-readable storage medium can be any electronic, magnetic, optical, or other physical storage device that can contain or store information such as executable instructions, data, etc. For instance, machine-readable storage media can be: RAM (Random Access Memory), volatile memory, non-volatile memory, flash memory, storage drives (such as hard disk drives), solid-state drives, any type of storage disk (such as optical discs, DVDs, etc.), or similar storage media, or combinations thereof.

[0117] The systems, devices, modules, or units described in the above embodiments can be implemented by a computer processor or entity, or by a product with a certain function. A typical implementation device is a computer, which can be a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email sending and receiving device, game console, tablet computer, wearable device, or any combination of these devices.

[0118] For ease of description, the above devices are described separately by function as various units. Of course, in implementing this application, the functions of each unit can be implemented in one or more software and / or hardware.

[0119] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, embodiments of this application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0120] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0121] Furthermore, these computer program instructions can also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in the process. Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0122] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0123] The above description is merely an embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. An image fusion method, characterized in that, This method is applied to electronic devices, including: Obtain first image data and second image data to be fused; the first image data is based on two different images, and the second image data is based on two different images. The first image data and the second image data are input into a trained target fusion network to fuse at least the first result and the second result to obtain fused image data. The first result refers to the result obtained by performing a specified operation on the feature value of each pixel in the first image data and the weight corresponding to that pixel in the first fusion weight matrix. The second result refers to the result obtained by performing a specified operation on the feature value of each pixel in the second image data and the weight corresponding to that pixel in the second fusion weight matrix. The first fusion weight matrix and the second fusion weight matrix are the model parameters trained by the target fusion network. The loss function used for training the target fusion network includes a loss function that reflects the X-ray fluoroscopic characteristics. ,Should Used to constrain the structural similarity between the fused feature image output by the fusion network to be trained and the two input training feature images during the training process.

2. The method according to claim 1, characterized in that, The first image data and the second image data are respectively a first image and a second image acquired by the acquisition device; or, The target fusion network is embedded between the encoder and decoder of the trained autoencoder network; The first image data is the first feature image extracted by the encoder after the first image acquired by the acquisition device is input to the encoder; The second image data is the second feature image extracted by the encoder after the second image acquired by the acquisition device is input to the encoder.

3. The method according to claim 1, characterized in that, The process of fusing at least the first result and the second result through a target fusion network to obtain fused image data includes: The feature values ​​of pixels at the same position in the first and second results are fused to obtain the third result; the first semantic feature is extracted from the third result by the first feature extractor in the target fusion network; The first image data and the second image data are spliced ​​at the channel level to obtain spliced ​​image data. The second semantic feature is extracted from the spliced ​​image data by the second feature extractor in the target fusion network. The first semantic feature and the second semantic feature are fused to obtain the fused image data.

4. The method according to claim 1, characterized in that, The target fusion network is trained through the following steps: The fusion network to be trained and the already deployed discriminator network are trained as an adversarial network; the fusion network to be trained is deployed between the encoder and decoder of the already trained autoencoder network, and the discriminator network is deployed after the decoder. During adversarial network training, if the loss function of the fusion network to be trained is... The loss function of the discriminator network must satisfy the corresponding fusion network loss requirements. If the corresponding discriminator network loss requirement is met, the current fusion network to be trained can be used as the target fusion network.

5. The method according to claim 4, characterized in that, The loss function This can be expressed by the following formula: ; in, and These are preset values. This represents a loss function used to reflect the characteristics of X-ray imaging. This refers to the output result obtained after inputting the fused image O into the discriminator network. The fused image O refers to the output result obtained after inputting the fused feature image O' into the decoder. The fused feature image O' refers to the output result obtained after inputting the training feature image O' into the decoder. ' and training feature images The input is fused into the fusion network to be trained, and the resulting output is the training feature image. 'Refers to training images The output of the encoder, the training feature image 'Refers to training images The output result obtained by inputting into the encoder; BCE() represents the binary cross-entropy loss used for binary classification.

6. The method according to claim 4, characterized in that, The loss function This can be expressed by the following formula: ; in, , and These are preset values. Indicates training images The output result obtained after inputting into the discriminator network Indicates training images The output result obtained after inputting into the discriminator network; This refers to the output result obtained after inputting the fused image O into the discriminator network. The fused image O refers to the output result obtained after inputting the fused feature image O' into the decoder. The fused feature image O' refers to the output result obtained after inputting the training feature image O' into the decoder. ' and training feature images The input is fused into the fusion network to be trained, and the resulting output is the training feature image. 'Refers to training images The output of the encoder, the training feature image 'Refers to training images The output result obtained by inputting into the encoder; BCE() represents the binary cross-entropy loss used for binary classification.

7. The method according to claim 5, characterized in that, The This can be expressed by the following formula: ； Represents the training feature image The similarity between 'and the fused feature image O' Represents the training feature image The similarity between 'and the fused feature image O' ; Represents the structural similarity index; When training feature images The similarity between 'and the fused feature image O' is the first set similarity, and the training feature image is... The similarity between 'and the fused feature image O' is set to the second similarity, and the transmission characteristics of the fusion network are optimal.

8. The method according to claim 6, characterized in that, The discriminator network is a spectral normalized discriminator network.

9. The method according to claim 2 or 4, characterized in that, The loss function used by the autoencoder network during training This can be expressed by the following formula: ; in, express and Pixel-level loss between These represent training images used to train the autoencoder network. Indicates will The output obtained by inputting the data into the decoder of the autoencoder network to be trained. This refers to The feature image obtained by the encoder in the autoencoder network to be trained; This can be expressed by the following formula: MAE represents the mean absolute error; express and Feature-level loss between express The feature image obtained after inputting into a third-party encoder, which is an encoder added during the training of the autoencoder network to assist in calculating the feature-level loss. This can be expressed by the following formula: , Indicates the mean squared error loss; , These are pre-set values.

10. An image fusion apparatus, characterized in that, This device is used in electronic devices, including: The obtaining unit is used to obtain first image data and second image data to be fused; the first image data is based on two different images, and the second image data is based on two different images. A fusion unit is used to input the first image data and the second image data into a trained target fusion network, so as to fuse at least the first result and the second result through the target fusion network to obtain a fused image; the first result refers to the result obtained by performing a specified operation on the feature value of each pixel in the first image data and the weight corresponding to that pixel in the first fusion weight matrix; the second result refers to the result obtained by performing a specified operation on the feature value of each pixel in the second image data and the weight corresponding to that pixel in the second fusion weight matrix; the first fusion weight matrix and the second fusion weight matrix are model parameters trained by the target fusion network; the loss function used for training the target fusion network includes a loss function that reflects the X-ray fluoroscopic characteristics. ,Should Used to constrain the structural similarity between the fused feature image output by the fusion network to be trained and the two input training feature images during the training process.

11. An electronic device, characterized in that, The electronic device includes: a processor and a machine-readable storage medium; The machine-readable storage medium stores machine-executable instructions that can be executed by the processor; The processor is used to execute machine-executable instructions to implement the method steps of any one of claims 1-9.

Images