Multi-energy X-ray image fusion method and device based on deep learning

Abstract

The present invention relates to a multi-energy X-ray image fusion method and device based on deep learning, collecting different energy X-ray images of different workpieces as training data sets; inputting the X-ray images of the training data sets into an encoder, training the encoder and decoder, and obtaining the trained encoder and decoder after the training network is stable; adding a fusion strategy combining channel attention and spatial attention based on fuzzy entropy between the trained encoder and decoder; inputting X-ray images of different energies into the trained encoder to extract features, and then using a fusion strategy combining channel attention and spatial attention based on fuzzy entropy to fuse feature maps, and the fused feature maps then enter the trained decoder to output the fusion results. Processing images by the method and device of the present invention can effectively reflect workpiece information and improve detection accuracy.

Description

Technical Field

[0001] The present invention relates to a multi-energy X-ray image fusion method and device based on deep learning. Background Art

[0002] Important components of complex structures play an irreplaceable role in aerospace, defense and industrial applications. Manufacturers need to strictly control the quality when manufacturing these workpieces, and X-rays are often regarded as a tool for quality inspection. Digital X-ray imaging technology can complete a series of tasks such as defect detection and internal structure analysis, but due to the constraints of workpiece structure and material, objects with large thickness ratios are overexposed and underexposed under a single-energy system, making it difficult to reflect comprehensive structural information. Specifically: in the X-ray image, when the thicker area of ​​the workpiece is displayed well, only a small part of the X-rays are absorbed in the thinner area, which may be overexposed; on the contrary, when the thinner part of the workpiece is displayed well, the thicker area cannot guarantee the remaining intensity after the X-rays pass through, which may appear as underexposure. Image fusion can fuse the information of different sensors or the same sensor under different imaging conditions, better improve the image resolution and clarity, and enhance and complement the relevant features of the image. Fusion of X-ray images under different energy irradiation can better solve the problem of coexistence of overexposure and underexposure during workpiece inspection. There are two important factors that determine the final result of image fusion: one is how to detect and extract information from the source image, and the other is to design appropriate rules to fuse the extracted feature information.

[0003] In the prior art, the public and popular image fusion methods can be divided into five categories: spatial domain processing algorithms, transform domain processing algorithms, sparse representation algorithms, deep learning algorithms, and composite algorithms. The spatial domain processing algorithm uses the pixels of the original image as the basic unit and directly processes the image. A typical example is the use of weighted average to obtain the fusion weight map. The processing speed is fast, but it is easy to introduce noise and it is difficult to adaptively determine the importance of each feature. The image fusion method in the transform domain uses the complexity of the calculation process in exchange for the retention of relatively more details. The block-based sparse representation algorithm fuses images through sparse coefficient features. The process is simple, but the algorithm is affected by the block strategy and block size, and is often interfered by unnecessary background information. The trained dictionary contains too many unnecessary features, and the fusion result has spatial discontinuity and sometimes contains black edges, which is not conducive to practical application. The method based on deep learning has many applications in infrared and visible light image fusion, medical image fusion, and multi-focus image fusion, reflecting good fusion effect and good robustness. However, there are limited public literature on multi-energy X-ray image fusion methods under deep learning. Most X-ray-based workpiece detection methods obtain transillumination sub-images of different thickness ranges by increasing the X-tube voltage, and obtain fusion results by weighted fusion of the sub-images. However, the weighted fusion process may cause the image depth range to far exceed the display capability of the equipment, making it impossible to fully display the structural information of the workpiece. Summary of the invention

[0004] The object of the present invention is to provide a multi-energy X-ray image fusion method and device based on deep learning, which can effectively reflect workpiece information and improve detection accuracy.

[0005] Based on the same inventive concept, the present invention has two independent technical solutions:

[0006] 1. A multi-energy X-ray image fusion method based on deep learning, a multi-energy X-ray image fusion method based on deep learning, comprising the following steps:

[0007] Step 1: Collect different energy X-ray images of different workpieces as training data sets;

[0008] Step 2: Input the X-ray image of the training data set into the encoder, train the encoder and decoder, and obtain the trained encoder and decoder after the training network is stable;

[0009] Step 3: Add a fusion strategy combining channel attention and fuzzy entropy-based spatial attention between the trained encoder and decoder;

[0010] Step 4: Input X-ray images of different energies into the trained encoder to extract features, and then use a fusion strategy that combines channel attention and spatial attention based on fuzzy entropy to fuse the feature maps. The fused feature maps then enter the trained decoder to output the fusion results.

[0011] Furthermore, in step 2, the encoder consists of a main branch and an auxiliary branch. The main branch first uses a 1×1 convolution layer to increase the number of feature channels of the input image, and then uses a 4-layer multi-scale convolution block to extract the global features of the image; the auxiliary branch uses a trainable edge detection operator to enhance the edge information of the input image to obtain a feature map of the same dimension as the main branch. The only difference between the auxiliary branch and the main branch is that the starting convolution layer of the main branch is replaced by a trainable edge detection operator.

[0012] Furthermore, in step 2, the features extracted by the main branch and the auxiliary branch of the encoder are added and then enter the decoder. At the end convolution layer of the decoder, the number of image channels is changed to a single channel, and the reconstructed image is output.

[0013] Further, in step 2, the edge detection operator used by the auxiliary branch is a Sobel operator, a Laplacian operator, a Canny operator or a LOG operator.

[0014] Furthermore, in step 2, both the main branch and the auxiliary branch are connected to a dense bypass, and the bypass is used to improve feature multiplexing capability.

[0015] Furthermore, in step 2, the multi-scale convolution block has four branches, and each convolution block consists of a convolution layer, an activation function, and a batch normalization layer.

[0016] Furthermore, in step 2, the activation function of the convolution block adopts Sigmoid function, Relu function, Tanh function, Softmax function or Leaky Relu function.

[0017] Furthermore, in step 2, the composite loss function guiding network training is composed of a weighted combination of an L1 loss function and an image block-based consistency loss function; the image block-based consistency loss function is used to calculate the L1 norm value loss of the local energy map of the input image and the local energy map of the output image, and the pixel points in the local energy map are calculated by averaging the sum of the squares of the differences between the neighborhood pixels and the intermediate pixels in an image block.

[0018] Furthermore, in step 3, the fusion strategy combining channel attention and fuzzy entropy-based spatial attention is specifically to fuse the feature maps using the channel attention module and the fuzzy entropy-based spatial attention module respectively, and take the average of the outputs of the two fusion modules as the feature map fusion result.

[0019] 2. A multi-energy X-ray image fusion device based on deep learning, used to execute the above method.

[0020] The present invention has the beneficial effects:

[0021] The present invention collects different energy X-ray images of different workpieces as training data sets; inputs the X-ray images of the training data sets into the encoder, trains the encoder and decoder, and obtains the trained encoder and decoder after the training network is stable; adds a fusion strategy combining channel attention and spatial attention based on fuzzy entropy between the trained encoder and decoder; inputs X-ray images of different energies into the trained encoder to extract features, and then uses the fusion strategy combining channel attention and spatial attention based on fuzzy entropy to fuse the feature map, and the fused feature map enters the trained decoder to output the fusion result. The present invention utilizes the powerful feature extraction ability of the neural network and the feature fusion strategy combining channel attention and spatial attention based on fuzzy entropy, and well fuses the effective information in the X-ray images of different energies of the workpiece while considering the complexity of the algorithm, and the fused image has rich edge details. The proposed algorithm also shows strong robustness to different fusion objects, overcomes the overexposure and underexposure problems of workpieces with uneven shapes and complex internal structures, and completely and clearly displays the internal information of complex components, which greatly improves the accuracy of internal defect detection and measurement of workpieces.

[0022] The encoder of the present invention is composed of a main branch and an auxiliary branch. The main branch first uses a 1×1 convolution layer to increase the number of feature channels of the input image, and then uses a 4-layer multi-scale convolution block to extract the global features of the image; the auxiliary branch uses a trainable edge detection operator to enhance the edge information of the input image to obtain a feature map of the same dimension as the main branch; the main branch and the auxiliary branch are both connected to a dense bypass, which is used to improve the feature reuse capability. The main branch of the present invention uses a 4-layer multi-scale convolution block to extract the global features of the image, and enhances the edge information of the image through the auxiliary branch, further ensuring the effective extraction of image features.

[0023] The fusion strategy of the present invention combining channel attention and fuzzy entropy-based spatial attention is specifically to fuse the feature map using the channel attention module and the fuzzy entropy-based spatial attention module respectively, and take the average of the outputs of the two fusion modules as the feature map fusion result. Through the above fusion strategy, the present invention further ensures that the effective information in the workpiece X-ray images of different energies is well fused. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 It is a schematic diagram of a multi-energy X-ray image fusion method based on deep learning of the present invention;

[0025] Figure 2 It is a schematic diagram of feature extraction by the encoder of the present invention;

[0026] Figure 3 Schematic diagram of a multi-scale convolution block in the encoder of the present invention;

[0027] Figure 4 It is a schematic diagram of a fusion strategy combining channel attention and fuzzy entropy-based spatial attention in the present invention;

[0028] Figure 5 is a schematic diagram of the channel attention module;

[0029] Figure 6 Schematic diagram of the spatial attention module based on fuzzy entropy. DETAILED DESCRIPTION

[0030] The present invention is described in detail below in conjunction with the various embodiments shown in the accompanying drawings, but it should be noted that these embodiments are not limitations of the present invention, and any equivalent transformations or substitutions in functions, methods, or structures made by ordinary technicians in the field based on these embodiments are all within the scope of protection of the present invention.

[0031] Embodiment 1:

[0032] Multi-energy X-ray image fusion method based on deep learning

[0033] like Figure 1 , Figure 2 As shown, the multi-energy X-ray image fusion method based on deep learning of the present invention includes the following steps:

[0034] Step 1: Collect X-ray images of different energies of different workpieces as training data sets.

[0035] The hyperparameters of the network are set, and the dataset images are scaled to a size suitable for the network input. In this embodiment, the input images are scaled to a size of 256×256.

[0036] Step 2: Input the X-ray image of the training data set into the encoder, train the encoder and decoder, and obtain the trained encoder and decoder after the training network is stable.

[0037] The encoder consists of a main branch and an auxiliary branch. The main branch first uses a 1×1 convolution layer to increase the number of feature channels of the input image, and then uses a 4-layer multi-scale convolution block to extract the global features of the image; the auxiliary branch uses a trainable edge detection operator to enhance the edge information of the input image to obtain a feature map of the same dimension as the main branch. The features extracted by the main branch and the auxiliary branch of the encoder are added and then enter the decoder. The convolution layer at the end of the decoder changes the number of image channels to a single channel and outputs a reconstructed image. The decoder structure is a complex network of U-Nets with nested depths, thereby realizing the fusion of feature maps with different semantics.

[0038] Both the main branch and the auxiliary branch are connected to a dense bypass, which is used to improve the feature reuse capability. The only difference between the auxiliary branch of the encoder and the main branch is that the starting convolution layer of the main branch is replaced by a trainable edge detection operator. The auxiliary branch adds the feature maps output by each layer of the multi-scale module to the feature maps of the corresponding layer of the main branch, thereby enhancing the edge information of the image for each layer of the feature map. The feature maps of each layer with added edge information enter the decoder to restore the image.

[0039] In order to make full use of the concept of multi-scale fusion, the convolution blocks of the encoder are all inception multi-scale feature extraction modules, and the overall network fully and effectively utilizes the multi-scale information and edge information of the features. Figure 3 As shown in the figure, the multi-scale convolution block has four branches, and each convolution block consists of a convolution layer, an activation function, and a batch normalization layer. The activation function of the convolution block uses Sigmoid function, Relu function, Tanh function, Softmax function, or Leaky Relu function. a×a is the size of the scaled input image; n is the number of convolution kernels of the learnable edge detection operator, which is also the number of convolution kernels in each convolution block; w×w is the size of the LOG template; s×s is the convolution kernel size; n, w, and s can all be adjusted according to the fusion effect. In addition, the network needs to use downsampling and upsampling to ensure that the size of the added feature maps is the same. Downsampling uses maximum pooling, and upsampling uses linear interpolation. Each branch uses a 1×1 convolution kernel to reduce the number of channels and the number of parameters. The first three branches have three different sizes of receptive fields, namely 1×1, 3×3 and 5×5. The third branch uses two 3×3 convolutions instead of 5×5 convolution kernels to further reduce the number of parameters while achieving the same effect. For the fourth branch, although the receptive field is the same as the second branch and the output of the same dimension is obtained, the output content is different, which increases the diversity of feature fusion. Finally, the different scale features extracted from the four branches are accumulated as the output of the multi-scale module. The multi-scale module increases the depth and width of the network and improves the generalization ability of the network.

[0040] The edge detection operator used in the auxiliary branch is Sobel operator, Laplacian operator, Canny operator or LOG operator. In this embodiment, the LOG (Gaussian Laplacian convolution) operator is used to first perform Gaussian smoothing on the image, and then convolve with the Laplacian operator. The commonly used LOG convolution template is 5×5 in size, and a learnable parameter α is defined. Each value in the LOG convolution template is multiplied by α, and α can be continuously updated as the network is trained. The 5×5 LOG convolution template is:

[0041]

[0042] Calculate the loss function, continuously train the network, improve the feature extraction ability of the codec network, and save the model after the network is stable. The composite loss function (L Compound ) is derived from the L1 loss function (L l1 ) and the image patch-based consistency loss function (L lpatch ) weighted combination, the formula is as follows:

[0043]

[0044] The image block-based consistency loss function is used to calculate the L1 norm value loss of the local energy map of the input image and the local energy map of the output image. The pixel points in the local energy map are calculated by summing the squares of the differences between the neighborhood pixels and the middle pixels in an image block and then averaging them.

[0045] like Figure 1 , Figure 2 As shown in the figure, during the training phase, the network only includes the encoder and decoder, but does not include a fusion strategy. Assuming that the input is a single-channel X-ray image of size a×a, the main branch of the encoder first uses a convolutional layer (containing n convolutional kernels of size 1×1) to increase the dimension of the input image and generate n feature maps of size a×a. After passing through the convolution block E12 (the number of output channels in the multi-scale module is n), n new feature maps of size a×a will be generated, and then after passing through the convolution block E22, n feature maps of size a / 2×a / 2 will be generated. By analogy, the last convolution block of this branch of the encoder outputs n feature maps of size a / 8×a / 8.

[0046] like Figure 1 As shown in the figure, the auxiliary branch of the encoder uses the edge detection operator of the auxiliary branch of the encoder (including n groups of w×w trainable edge detection operators, where w must be an odd number) to extract the edge information of the image and generate n feature maps of size a×a. The remaining operations are the same as the main branch. The feature maps of the corresponding depths of the two branches are added and output to the decoder. The decoder contains decoder subnets of different depths. Convolution block D31 outputs n feature maps of size a / 4×a / 4, convolution blocks D21 and convolution blocks D22 output n feature maps of size a / 2×a / 2, convolution blocks D11, convolution blocks D12 and convolution blocks D13 output n feature maps of size a×a, and the convolution layer at the end of the decoder outputs an X-ray map of channel size a×a.

[0047] The implementation methods of initializing the weight parameters of the convolution kernel are: random initialization, Gaussian initialization, etc. The bypass connection module in the network design uses a skip connection method to connect two convolutional layers with a certain interval. The size of the interval and the number of bypass connections can be determined and adjusted according to the number of convolutional layers. The initialization learning rate and the number of training times of the network model training module can be determined and adjusted according to the convergence of the network (in this embodiment, the learning rate is initialized to 10-4 , the total number of training times is set to 10 times). The network parameter update of the network model training module is implemented by: stochastic gradient descent algorithm (SGD algorithm), Adam algorithm, etc. During the training process, one or more of the learning rate, number of convolution kernels, size of convolution kernels, weights, number of network layers, etc. can be continuously adjusted according to the convergence of the network and the fusion results.

[0048] Step 3: Add a fusion strategy combining channel attention and fuzzy entropy-based spatial attention between the trained encoder and decoder.

[0049] In the fusion stage, a specific fusion strategy is added to each layer (four layers in total) between the trained encoder and decoder as a complete network. K X-ray images of different energies to be fused are input into the model. Since the number of input images is K, the four-layer multi-scale Inception structure of the encoder will output K groups of feature maps. The fusion strategy of each layer is to fuse the K groups of feature maps of the corresponding layer, and then input the fused feature maps into the decoder to obtain the fused image.

[0050] like Figure 4 As shown, the fusion strategy combining channel attention and fuzzy entropy-based spatial attention is to fuse the feature maps using the channel attention module and the fuzzy entropy-based spatial attention module respectively, and take the average of the outputs of the two fusion modules as the feature map fusion result.

[0051] For the input K groups of feature maps Fusion R W×H×C The length, width and number of channels of the feature map are W, H and C respectively. is the final fusion result of the K groups of feature maps at the mth layer, as shown in formula (1):

[0052]

[0053] like Figure 5 As shown, the channel attention module uses average pooling to perform K group feature maps Calculate the respective channel attention coefficients Normalize the attention parameter of the current feature map by dividing it by the sum of all attention parameters As shown in formula (2). Then multiply the feature map with the corresponding attention coefficient to form a new feature map All new feature maps are accumulated as the result of the channel attention fusion strategy. As shown in formula (3).

[0054]

[0055]

[0056] Where m∈{1,2,3,4},k∈{1,2,…,K} refers to the kth input image, K is the total number of fused images, and c refers to the feature map and attention coefficient The cth channel of .

[0057] like Figure 6 As shown in the figure, the spatial attention module based on fuzzy entropy first analyzes K groups of feature maps Calculate the respective fuzzy entropy graph Specifically, according to formula (4), the fuzzy entropy map corresponding to each channel in each set of feature maps is calculated.

[0058]

[0059] Where (x, y, c)∈R W×H×C , N x,y,c is the 3×3 neighborhood centered at (x,y) in the c-th channel feature map, (i,j,c)∈N x,y,c , is the membership degree of the neighborhood pixel centered at (x, y) in the c-th channel feature map, and the calculation formula is:

[0060]

[0061] Normalize the fuzzy entropy map of the current feature map by dividing it by the sum of all fuzzy entropy maps to get As shown in formula (6), it is used as the attention coefficient of the spatial attention module. Then the feature map is multiplied by the corresponding attention coefficient to form a new feature map And accumulate K groups of new feature maps to get the result of the spatial attention fusion strategy As shown in formula (7).

[0062]

[0063]

[0064] Where m∈{1,2,3,4} refers to the mth layer fusion strategy, k∈{1,2,…,K} refers to the kth input image, and K is the total number of fused images.

[0065] Step 4: Input X-ray images of different energies into the trained encoder to extract features, and then use a fusion strategy that combines channel attention and spatial attention based on fuzzy entropy to fuse the feature maps. The fused feature maps then enter the trained decoder to output the fusion results.

[0066] Embodiment 2:

[0067] Multi-energy X-ray image fusion device based on deep learning

[0068] The multi-energy X-ray image fusion device based on deep learning is used to execute the above-mentioned multi-energy X-ray image fusion method based on deep learning.

[0069] The series of detailed descriptions listed above are only specific descriptions of feasible implementation methods of the present invention. They are not intended to limit the scope of protection of the present invention. Any equivalent implementation methods or changes that do not deviate from the technical spirit of the present invention should be included in the scope of protection of the present invention.

[0070] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above and that the present invention can be implemented in other specific forms without departing from the spirit or essential features of the present invention. Therefore, the embodiments should be considered exemplary and non-restrictive in all respects, and the scope of the present invention is defined by the appended claims rather than the above description, and it is intended that all changes falling within the meaning and scope of the equivalent elements of the claims be included in the present invention.

Claims

1. A multi-energy X-ray image fusion method based on deep learning, characterized in that: The steps include: Step 1: Collect different energy X-ray images of different workpieces as training data sets; Step 2: Input the X-ray image of the training data set into the encoder, train the encoder and decoder, and obtain the trained encoder and decoder after the training network is stable; Step 3: Add a fusion strategy combining channel attention and fuzzy entropy-based spatial attention between the trained encoder and decoder; Step 4: Input X-ray images of different energies to the trained encoder to extract features, and then use the fusion strategy combining channel attention and spatial attention based on fuzzy entropy to fuse the feature maps. The fused feature maps then enter the trained decoder to output the fusion results. In step 2, the encoder consists of a main branch and an auxiliary branch. The main branch first uses a 1×1 convolution layer to increase the number of feature channels of the input image, and then uses a 4-layer multi-scale convolution block to extract the global features of the image; the auxiliary branch uses a trainable edge detection operator to enhance the edge information of the input image to obtain a feature map of the same dimension as the main branch. The only difference between the auxiliary branch and the main branch is that the trainable edge detection operator is used to replace the starting convolution layer of the main branch; In step 2, the features extracted by the main branch and the auxiliary branch of the encoder are added and then enter the decoder. At the end of the decoder, the convolutional layer changes the number of image channels to a single channel and outputs the reconstructed image. In step 2, both the main branch and the auxiliary branch are connected to a dense bypass, and the bypass is used to improve the feature multiplexing capability; In step 3, the fusion strategy combining channel attention and fuzzy entropy-based spatial attention is to fuse the feature maps using the channel attention module and the fuzzy entropy-based spatial attention module respectively, and take the average of the outputs of the two fusion modules as the feature map fusion result; In step 3, the channel attention module uses average pooling K The group feature maps calculate their respective channel attention coefficients, divide the attention coefficient of the current feature map by the sum of all attention coefficients to normalize it to get the current attention coefficient, and then multiply the feature map by the corresponding attention coefficient to form a new feature map. All new feature maps are accumulated as the output result of the channel attention fusion strategy; In step 3, the fuzzy entropy-based spatial attention module first K The fuzzy entropy map of each group feature map is calculated, and the fuzzy entropy map of the current feature map is divided by the sum of all fuzzy entropy maps for normalization to obtain the current attention coefficient; then the feature map is multiplied by the corresponding attention coefficient to form a new feature map, and K The new feature maps are accumulated to obtain the output result of the spatial attention fusion strategy.

2. The multi-energy X-ray image fusion method based on deep learning according to claim 1, characterized in that: In step 2, the edge detection operator used by the auxiliary branch is a Sobel operator, a Laplacian operator, a Canny operator or a LOG operator.

3. The multi-energy X-ray image fusion method based on deep learning according to claim 1, characterized in that: In step 2, the multi-scale convolution block has four branches, and each convolution block consists of a convolution layer, an activation function, and a batch normalization layer.

4. The multi-energy X-ray image fusion method based on deep learning according to claim 3 is characterized in that: In step 2, the activation function of the convolution block uses the Sigmoid function, Relu function, Tanh function, Softmax function or Leaky Relu function.

5. The multi-energy X-ray image fusion method based on deep learning according to claim 1, characterized in that: In step 2, the composite loss function guiding network training is composed of a weighted combination of an L1 loss function and an image block-based consistency loss function; the image block-based consistency loss function is used to calculate the L1 norm value loss of the local energy map of the input image and the local energy map of the output image, and the pixel points in the local energy map are calculated by averaging the sum of the squares of the differences between the neighborhood pixels and the middle pixels in an image block.

6. A multi-energy X-ray image fusion device based on deep learning, characterized in that: Used to execute the method according to any one of claims 1 to 5.

Images