A CNN-Transformer Multi-Spectrum Digital Ray Image Fusion Method

Abstract

This invention provides a CNN-Transformer multi-energy spectral digital X-ray image fusion method, belonging to the field of image processing technology. It includes: receiving DR images obtained at different energies; preprocessing the input images; inputting the preprocessed images into a dual-branch feature encoder to extract local, global, and deep features; performing feature enhancement and fusion processing on the images, with an enhancement block module enhancing the extracted features; a cross-attention fusion decoder module fusing CNN and Transformer features using a cross-attention mechanism; decoding and reconstructing the images, inputting the fused features into a decoder module, and further optimizing feature representation through an adaptive attention fusion mechanism; simultaneously, upsampling through multiple ConvBlock layers to restore image details; and training using a composite loss function. This invention effectively solves the problem of overexposure and underexposure caused by differences in workpiece thickness due to single X-ray energy.

Description

Technical Field

[0001] This invention provides a CNN-Transformer multi-energy spectrum digital ray image fusion method, belonging to the field of image processing technology. Background Technology

[0002] Industrial development relies heavily on various non-destructive testing (NDT) technologies, including ultrasonic testing, X-ray testing, magnetic particle testing, eddy current testing, and penetrant testing. Current NDT systems are insufficient to meet the demands of industrial transformation, and the need for novel NDT methods is growing across industries. Therefore, developing new NDT methods and optimizing existing ones has become an urgent issue. For quality inspection of complex industrial components, digital X-ray imaging technology, with its high efficiency, real-time performance, high quality, flexibility, safety, and environmental friendliness, has become an indispensable core inspection tool. It can accurately and efficiently detect internal defects, ensuring product quality and structural safety. These complex structural components often have a large thickness range in the transmission direction, resulting in significant differences in the equivalent thickness during X-ray transmission. However, the dynamic range of X-ray imaging systems is limited, and single-energy X-rays cannot adapt to different thickness differences. In actual inspection, overexposure and underexposure often occur, leading to the inaccurate capture and identification of internal defects, thus posing significant safety hazards. To solve this problem, introducing a multi-energy spectral fusion method for digital X-ray images is crucial.

[0003] It is worth noting that digital X-ray imaging multi-spectral fusion technology, as an advanced non-destructive testing technology, can effectively distinguish different materials with similar densities that are difficult to identify in traditional single-energy imaging, significantly enhance the contrast of internal defects, and suppress noise, thereby enabling more accurate identification and evaluation of the internal structure and material of the inspected workpiece.

[0004] In 2016, Wang Quan et al. [2016, Wang Quan, Nie Rencan, Jin Xin, Zhou Dongming, He Kangjian, Yu Jiefu, "Image Fusion Algorithm Based on Laplacian Pyramid and PCNN-SML"] proposed an image fusion algorithm based on Laplacian pyramid and PCNN-SML (Pulse Coupled Neural Network - Sum of Modified Laplacian). This algorithm first decomposes the source image using a Laplacian pyramid at multiple scales. Then, it processes each scale of the decomposed image using PCNN to obtain a neuron firing frequency map describing feature clustering. Based on the local SML of the firing frequency map, it achieves the fusion of the LP (Laplace Pyramid Algorithm) decomposition of the source image at each scale. Finally, it achieves the fusion of the focused image through inverse LP transform. This method involves iterative processing of PCNN and multi-level pyramid decomposition and reconstruction, resulting in high computational cost and algorithm complexity, leading to low operating efficiency and inability to meet real-time requirements. Furthermore, the model's parameter settings are highly dependent; different parameter settings significantly affect the fused image, impacting the algorithm's robustness and adaptability.

[0005] Experiments have shown that the image fusion algorithm based on Laplacian pyramid and PCNN-SML proposed by Wang Quan, Nie Rencan, Jin Xin, and others outperforms traditional image fusion methods and can improve the quality of fused images to a certain extent. However, it also has the following drawbacks:

[0006] 1) The algorithm is highly complex and computationally intensive. The PCNN iterative calculation and Laplacian pyramid multi-scale decomposition and reconstruction processes involved in the algorithm are computationally intensive, resulting in low model efficiency and difficulty in meeting real-time requirements.

[0007] 2) The model has many parameters and is complex to set. The model parameter settings rely on the experience of the detection personnel. For example, the link strength and decay coefficient of PCNN and the threshold in SML need to be adjusted manually, which affects the robustness and adaptability of the model to a certain extent.

[0008] 3) The selection of thresholds is quite sensitive during image fusion. Slightly improper settings may lead to loss of image details and affect the quality of the fused image. Summary of the Invention

[0009] This invention provides a CNN-Transformer multi-energy spectral digital ray image fusion method, the technical problems to be solved include:

[0010] 1) Optimize the shortcomings and limitations of traditional image fusion algorithms, solve the problem of overexposure and underexposure when a single X-ray energy is used to irradiate workpieces of different thicknesses, improve the clarity of radiographs, and prevent the omission of small defects in complex workpieces.

[0011] 2) To address the issues of high computational cost and complexity in existing image fusion algorithms, this technical solution incorporates a CNN module and a Transformer module. The fusion process propagates forward in a single step, significantly improving the algorithm's efficiency and enabling it to better meet the real-time requirements of industrial inspection.

[0012] 3) Addressing the issues of high parameter dependence and weak adaptability in existing image fusion technologies, this paper utilizes a fusion module to extract features from multi-spectral inputs and dynamically and adaptively determine how to fuse information. This enables end-to-end automatic learning without manual parameter tuning, significantly improving the adaptability and generalization of the algorithm model.

[0013] 4) Optimized feature representation, achieving organic fusion of local and global features. The CNN-Block module in the hybrid encoder can quickly and effectively capture local details and spatial features of the image. The Transformer module models the connections between all regions of the image through a self-attention mechanism, solving the problem of traditional algorithms lacking a global perspective and better handling large-scale structures or complex backgrounds. This achieves organic fusion of local and global information, greatly improving the accuracy of complex defect detection.

[0014] 5) Applying a joint loss function effectively balances pixel accuracy and visual quality. The fusion result is partially guaranteed to be close to the ideal output in terms of pixel values, preserving image details and edges, and ensuring the fidelity of the fusion result in terms of pixel intensity. The fusion method partially guarantees that the fusion result is optimal in terms of structural similarity, which conforms to human visual perception, making the fused image look more natural and clearer. This process makes the merged image closer to the real image, enhancing the realism of the merged image.

[0015] The specific technical solution is as follows:

[0016] A CNN-Transformer multi-energy spectral digital ray image fusion method includes the following steps:

[0017] (1) The system receives DR images obtained under different energies. The preprocessing module adjusts the input images to a uniform size of 256×256 pixels to ensure that the size of the input images remains consistent. At the same time, the pixel values ​​are normalized to the range of [0, 1].

[0018] (2) The preprocessed image is input into the dual-branch feature encoder: the convolutional neural network module is responsible for extracting local features, and the Transformer module is responsible for extracting global features. In the feature extraction module, the two modules work together to extract rich and complementary deep feature representations from the preprocessed image. Feature extraction is performed simultaneously through two parallel paths. The convolutional neural network module has a structure of four cascaded ConvBlocks. This module focuses on the local features of the image and can perform layer-by-layer scanning of local details in the obtained image to extract detailed features such as texture, edges, and local contrast. The Transformer module focuses on the global features of the image. Its structure includes multi-head self-attention MSA and multilayer perceptron MLP, and uses layer normalization technology to stabilize training. It is good at capturing long-distance dependencies and global contextual information of the image and establishes the connection between any two positions in the image through the self-attention mechanism.

[0019] (3) The image undergoes feature enhancement and fusion processing. The enhancement block module enhances the extracted features. The cross-attention fusion decoder module uses a cross-attention mechanism to fuse features from CNN and Transformer.

[0020] (4) The image is decoded and reconstructed. The fused features are input into the decoder module, and the feature representation is further optimized through an adaptive attention fusion mechanism. At the same time, the image details are restored by upsampling through multiple ConvBlock layers.

[0021] (5) Training using a composite loss function: total loss ;

[0022] in, Representing structural similarity loss, it minimizes the structural differences between the fused image and the target image; This represents feature-perceptual loss, which can improve the realism of the final image; L1 loss, also known as mean absolute error (MAE), enables the fused image to generate sharper edges and is less prone to over-smoothing. The three hyperparameters α, β, and γ are used to balance the weights of the various losses and control their contribution to the overall optimization.

[0023] The technical effects of the present invention are as follows:

[0024] The quality of the fused image obtained by the CNN-Transformer fusion method is superior to other traditional image fusion techniques in all aspects. The invention effectively solves the problem of overexposure and underexposure caused by the difference in workpiece thickness due to single X-ray energy, greatly improves the detection rate of small defects in complex workpieces, and greatly promotes the development and progress of non-destructive testing technology. Attached Figure Description

[0025] Figure 1 This is a framework diagram of the CNN-Transformer fusion method in an embodiment;

[0026] Figure 2 This is a diagram showing the composition of the experimental setup for an embodiment.

[0027] Figure 3 The following is a diagram showing the fusion results of each algorithm artifact 1 in the embodiment;

[0028] Figure 4 The following is a diagram showing the fusion results of each algorithm artifact 2 in the embodiment;

[0029] Figure 5 The following is a diagram showing the fusion results of the various algorithm artifacts 3 in the embodiment;

[0030] Figure 6 The following is a diagram showing the fusion results of the various algorithm artifacts in the embodiment;

[0031] Figure 7 The following is a diagram showing the fusion results of the various algorithm artifacts 5 in the embodiment;

[0032] Figure 8 The following is a diagram showing the fusion results of the various algorithm artifacts 6 in the embodiment;

[0033] Figure 9 A comparison of edge densities for different image fusion algorithms in the examples;

[0034] Figure 10 A comparison of the sharpness of different image fusion algorithms used in the embodiments;

[0035] Figure 11 Contrast comparison of different image fusion algorithms used in the embodiments;

[0036] Figure 12 A comparison of the overall quality of different image fusion algorithms in the examples;

[0037] Figure 13 The computation time and PSNR value as a function of sampling steps are shown in the example.

[0038] Figure 14 The evolution of PSNR values ​​during model training is shown in the example.

[0039] Figure 15 The evolution of SSIM values ​​during model training is shown in the example. Detailed Implementation

[0040] The specific technical solutions of the present invention will be described in conjunction with the accompanying drawings and embodiments.

[0041] A CNN-Transformer multi-energy spectral digital ray image fusion method includes the following steps:

[0042] The CNN-Transformer fusion system framework used in this embodiment is as follows: Figure 1 As shown, this system is a typical encoder-decoder deep learning model. This algorithm can fuse images obtained under different X-ray energies to generate a more comprehensive image with clearer defect information, thereby improving the quality of the obtained image, reducing the chance of missing minute defects in complex workpieces, and ultimately enhancing the accuracy of industrial non-destructive testing.

[0043] The specific steps of the image fusion algorithm are as follows:

[0044] (1) The system receives DR images obtained at different energies (low energy, medium energy, high energy). The preprocessing module adjusts the input images to a uniform size of 256×256 pixels to ensure that the size of the input images remains consistent. At the same time, the pixel values ​​are normalized to the range of [0, 1], which helps to improve the stability and convergence speed of model training, so as to facilitate the next step of processing.

[0045] (2) Input the preprocessed image into the dual-branch feature encoder:

[0046] The convolutional neural network module is responsible for extracting local features (convolutional layers). The Transformer module is responsible for extracting global features (self-attention mechanism). In the feature extraction module, the two modules work together to extract rich and complementary deep feature representations from the preprocessed image. Feature extraction is performed simultaneously through two parallel paths.

[0047] The Convolutional Neural Network (CNN) module consists of four cascaded ConvBlocks. This module focuses on local image features, performing layer-by-layer scanning of local details to efficiently extract texture, edges, local contrast, and other detailed features, which is particularly crucial for capturing minute defects (such as cracks and pores). The Transformer module focuses on global image features, employing a structure that includes Multi-Head Self-Attention (MSA) and Multilayer Perceptron (MLP), and using techniques such as LayerNorm for stable training. It excels at capturing long-range dependencies and global contextual information in images, establishing connections between any two locations in the image through a self-attention mechanism. The two parallel sub-modules operate simultaneously, complementing each other to capture minute defect information while maintaining the overall integrity of the image.

[0048] (3) The image is enhanced and fused using feature enhancement. The enhancement block module enhances the extracted features. The cross-attention fusion decoder module uses a cross-attention mechanism to fuse features from CNN and Transformer, combining local and global information to greatly improve the image fusion effect.

[0049] (4) The image is decoded and reconstructed. The fused features are input into the decoder module, and the feature representation is further optimized through an adaptive attention fusion mechanism. At the same time, the image details are restored by upsampling through multiple ConvBlock layers.

[0050] (5) In order for the system to produce the optimal fused image, the system is trained using a composite loss function: total loss .

[0051] in, This represents structural similarity loss, which is often used to measure the similarity between two images in terms of structure, brightness, and contrast. It preserves the overall structure and texture of the image and avoids generating results that are too blurry or structurally distorted. This represents feature-aware loss, which is often used to compare the differences between the generated image and the target image in the feature space, rather than just pixel-level differences. It can better capture semantic information and high-level visual features, and helps to generate visually more natural images. L1 loss, also known as mean absolute error (MAE), enables the fused image to generate sharper edges and is less prone to over-smoothing. The three hyperparameters α, β, and γ are used to balance the weights of the various loss parameters, controlling their contribution to the overall optimization. In experiments, different aspects of image quality can be optimized by adjusting the values ​​of these three coefficients.

[0052] In summary, the original image undergoes a complete process of preprocessing → bi-branch feature extraction → cross-modal feature fusion → decoding and reconstruction → loss optimization, ultimately outputting a fused image. This improves the ability to display minute defects in complex workpieces, and significantly enhances the overall quality of the fused image.

[0053] The image quality of the final fused image was compared and evaluated from both subjective and objective perspectives. The objective evaluation assessed the quality of the fused image using 13 evaluation indicators across 4 aspects.

[0054] (1) Subjective evaluation

[0055] Subjective evaluation refers to an observer's judgment and interpretation of image quality, defect features, and anomalies based on their accumulated professional experience, visual perception, and industry intuition. In subjective evaluation, the observer's visual perception becomes a crucial basis for judging the quality of the fused image. By intuitively experiencing aspects such as image sharpness, contrast, and color reproduction, the algorithm's visual performance can be evaluated. This method is highly intuitive and reflects the aesthetic appeal of the fusion result; however, this process is significantly influenced by the observer's abilities, psychological state, and even environmental factors. Therefore, conclusions obtained from subjective evaluation may vary from person to person, making the introduction of objective evaluation particularly important.

[0056] (2) Objective evaluation

[0057] Objective evaluation refers to the analysis and judgment of image quality, defect features, or abnormal performance based on quantifiable, verifiable, and repeatable physical parameters or standardized rules. Objective evaluation uses a series of quantitative indicators to accurately assess the performance of fusion algorithms. These indicators can reflect the performance of the fused image in terms of information content, edge details, and structure preservation from different perspectives, greatly avoiding subjectivity. However, it is worth noting that these objective evaluation indicators may have limitations under certain specific image content and features, and cannot be completely equated with the intuitive perception of the fusion result by the human eye. Therefore, when evaluating the quality of fused images, a combination of subjective and objective evaluation methods should be adopted. Objective evaluation indicators are shown in Table 1.

[0058] Table 1 Objective Evaluation Indicators

[0059]

[0060] First, the ↑ / ↓ arrow direction indicates whether a larger value (↑) or a smaller value (↓) is better. For example, PSNR↑ indicates a higher value and better image quality; NRMSE↓ indicates a lower value and smaller error.

[0061] Secondly, a reference metric indicates that an ideal or realistic reference image is needed as a standard for comparison. No reference metric means that no reference image is required; the fused image itself is analyzed and evaluated directly. This is more practical in real-world applications.

[0062] Finally, different categories of evaluation metrics are used to assess different aspects of the quality of the fused images.

[0063] Information theory metrics are used to measure the amount of information contained in an image or the degree of information retention. Among them, EN (Information Entropy) measures the richness of image information. A higher entropy value indicates a more dispersed gray-level distribution, containing more information and richer details. MI (Mutual Information) and NMI (Normalized Mutual Information) measure the amount of information shared between the fused image and the source image. A higher MI / NMI value indicates that the fused image retains more information from the source image. PSNR (Peak Signal-to-Noise Ratio) measures the pixel fidelity of the fused image relative to the reference image. A higher PSNR indicates smaller pixel errors and less image distortion. NRMSE (Normalized Root Mean Square Error) is another expression of PSNR, also based on pixel errors. A smaller value indicates a smaller error between the fused image and the reference image. SD (Standard Deviation) measures the dispersion of image contrast and gray-level distribution. A larger standard deviation indicates a wider range of gray-level values ​​and potentially stronger contrast.

[0064] Image feature metrics are used to measure image sharpness, detail, and activity. Among them, AG (average gradient) reflects the sharpness and richness of texture detail. The gradient represents the rate of change of pixel values; a larger average gradient indicates sharper image edges, clearer details, and more pronounced subtle texture variations. A higher SF (spatial frequency) value indicates a sharper overall image with richer details.

[0065] Correlation and structure metrics assess image quality from a structural or correlation perspective. Among them, SCD (Spectral Correlation Difference) measures the spectral (color, grayscale distribution) correlation between the fused image and the source image in the frequency domain. A higher value indicates that the fused image inherits the spectral characteristics of the source image better. SSIM and MS-SSIM (Structural Similarity) measure the similarity between two images in terms of brightness, contrast, and structure. MS-SSIM is a more comprehensive, multi-scale version. NRSS (No-Reference Structural Similarity) assesses the sharpness and structural integrity of an image itself.

[0066] Human perception metrics directly simulate the human visual system for evaluation. Among them, VIF (Visual Information Fidelity) measures how much visual information from the reference image is faithfully preserved in the fused image. It takes into account visual characteristics such as contrast sensitivity and channel decomposition. A high VIF value means that the fused image is not only similar in data, but also contains more effective visual information and appears more realistic to the human eye.

[0067] This embodiment employs a digital X-ray imaging system, which mainly consists of an X-ray source, a digital flat panel detector, a rotating stage, and a computer processing module. The X-ray source emits a beam that penetrates the workpiece placed on the rotating stage. During penetration, the X-rays undergo varying degrees of photon absorption and scattering due to the workpiece's internal structure (such as density differences, thickness variations, and defects), resulting in attenuated signals. The attenuated X-rays are received by the flat panel detector behind the workpiece. The detector directly converts the X-ray photons into electrical signals, which are then converted into digital grayscale images via an analog-to-digital converter. The computer workstation processes the received data, ultimately enabling visualization of the workpiece's internal structure and detection of internal defects. The basic components of the experimental setup are as follows: Figure 2 As shown.

[0068] To conduct a more comprehensive comparative analysis of the proposed multi-exposure image fusion algorithm, this embodiment fuses acquired X-ray image sequences using seven advanced fusion algorithms published in recent years. The fused images obtained by different algorithms are then compared subjectively and objectively based on various evaluation metrics. These algorithms include GF, GD, FPM, GFF, PMEF, MEF-AW, DSIFT-EF, MEFOpt, ExposureFusion, MTI, DEEF, PWA, SPD-MEF, FMMEF, DEM, MSPD-MEF, and EMEF. Subjective evaluation compares and assesses the images obtained by the proposed CNN-Transformer fusion method with those obtained by other image fusion methods in terms of image sharpness, contrast, detail preservation, defect display, image depth, and overall visual appeal.

[0069] The fusion results of processing the source image of workpiece 1 using different algorithms are as follows: Figure 3As shown. Compared to other algorithms, the fusion effect of the MTI method, GT method, MEFOpt method and the CNTrans method proposed in this embodiment is significantly better. The fused image obtained by the CNTrans method proposed in this embodiment has the most obvious layer division on the workpiece. It can clearly see the division of different thicknesses of workpiece 1 and has high clarity. The edge details are clear and rich, and the structure of workpiece 1 is completely preserved. The defect detection rate is greatly improved compared with the source image. The image obtained by the LE method has too much black overall, indicating overexposure and poor overall image contrast. The right image is almost completely black, with blurred details, and no defects can be detected at all; only a small part of the left side can detect defects, but the contrast is insufficient, and the defects cannot be fully displayed. The fused images obtained by the HE method and FMMEF method both show underexposure on the left side. The overall color of the film is very pale, white and gray, with very low image contrast. Defect details are almost not displayed, and the overall appearance of the film is poor. The fused image obtained by the Deepfuse method has poor clarity and contrast. The obtained defect image is very coarse, lacks detailed information, and it is almost impossible to distinguish defect information from workpiece information, so it cannot be applied to actual inspection. The fused images obtained by the MTI, Exposure Fusion, PWA, IFCNN, GT, and MEFOpt methods lack depth and cannot accurately display the thickness variation information of the workpiece. Among them, the IFCNN method produces the worst contrast in the fused image, almost failing to reveal defect information. The fused images obtained by the DEM, EMEF, and SwinFusion methods have stronger depth and better preservation of edge details, but the contrast between the defect image and the background is insufficient, resulting in incomplete and unclear display of defect information, which may lead to missed defects in actual inspection.

[0070] In summary, the CNN-Transformer fusion method proposed in this embodiment yields the best overall quality fused image.

[0071] The fusion results of processing the two source images of the workpiece using different algorithms are as follows: Figure 4 As shown in the figure, the CNTrans method proposed in this embodiment produces the best fused image clarity, rich image layering, and a perfect balance between overall workpiece information and local detail information with natural transitions, resulting in a good overall image quality. It achieves the best balance among clarity, contrast, local detail, layering, and defect display capabilities. The MTI method produces a fused image with an overall white color, indicating underexposure, which may lead to missed detections of defects in the middle of the workpiece. Compared to the CNTrans method, other algorithms produce fused images with more obvious overall contrast, but the contrast in the threaded area of ​​the workpiece is poor, with insufficient detail information, unclear layering, and unclear thread details, which may introduce certain errors in actual inspection.

[0072] The fusion results of processing the three source images of the workpiece using different algorithms are as follows: Figure 5 As shown. After fusing the source images using the following different algorithms, observe the same grid-like region of the workpiece. Figure 6 It can be observed that the images obtained by the ExposureFusion and MTI methods have low black levels, an overall grayish color, and extremely blurry grid edge details, resulting in poor overall image clarity. The common problems of the DEM, FMMEF, FPM, DeepFuse, SwinFusion, GFF, and U2Fusion methods are poor grid contrast in the central part of the image and unclear grid overlap areas. Information loss occurs on the left side of all images, posing a significant risk of missed defect detection. The edge details on the right side of the image lack clarity, resulting in weak overall visual impact and a lack of depth. The fused images obtained by the EMEF and GT methods show some improvement in overall appearance compared to the above methods, but the left side of the image is still relatively blurry, and the edge detail representation is insufficient. Through comparative evaluation, the final image obtained by the CNN-Transformer fusion method proposed in this embodiment is slightly superior in terms of image contrast, clarity, detail preservation, defect display, image depth, and overall appearance.

[0073] The fusion results of processing the four source images of the workpiece using different algorithms are as follows: Figure 6 As shown. After fusing the source images using the following different algorithms, the same striped area of ​​the workpiece was observed. The HE method resulted in an image that was too white, indicating underexposure and failing to display any information about the workpiece or defects. The LE, MEFOpt, and PWA methods had the main problem of high background darkness, which was not conducive to the detection of small defects. The MTI, Deepfuse, IFCNN, and U2Fusion methods resulted in images with high background grayscale, insufficient contrast between the striped information of the workpiece and the background information, and unclear stripe edges. The GFF method resulted in a fused image in which the edges of two stripes merged and were indistinguishable in the middle area. In the fused image obtained by the Swin Fusion method, small ridges appeared at the edges of each stripe, which would greatly increase the likelihood of misjudgment in actual inspection. As can be seen from the image comparison, the fused image obtained by the GT method is most similar to the final image obtained by the CNN-Transformer fusion method proposed in this embodiment. The image contrast, clarity, detail display capability, and overall visual quality are all good. However, the blackness of the background area in the middle of the image obtained by the GT method is larger than that of the background on both sides, resulting in uneven overall background blackness.

[0074] The fusion results of processing the 5 source images of the workpiece using different algorithms are as follows: Figure 7As shown. A significant characteristic of the final images obtained by the DEM, FMMEF, FPM, MEFAW, PMEF, PWA, and SPD-MEF methods is their overall high image blackness, which obscures internal workpiece details and poses a risk of missed detection in actual inspection. The fused images obtained by the DeepFuse, DSIFT-EF, ExposureFusion, and PMEF methods have a large overall grayscale, blurred edge details, and low image contrast, requiring high levels of visual acuity and experience from the observer. The final image obtained by the MTI method is generally white, with extremely blurred information around the workpiece and insufficient detail, resulting in a high risk of missed defect detection and making it unsuitable for practical inspection. The fused images obtained by the GF and GFF methods have uneven overall blackness, and their overall visual quality is inferior to the image obtained by the CNN-Transformer fusion method proposed in this embodiment. Specifically, the image obtained by the GF method has high blackness in the center but low blackness at the surrounding edges. The image obtained by the GFF method has low blackness in most areas and high blackness in some local areas, affecting the ability to display defects.

[0075] The fusion results of processing the 6 source images of the workpiece using different algorithms are as follows: Figure 8 As shown, the image obtained by the HE method suffers from underexposure in the upper left corner, while the image obtained by the LE method has overall high blackness and overexposure. Both methods carry the risk of missed detections and false positives. Compared to the image obtained by the CNN-Transformer fusion method proposed in this embodiment, the images obtained by other methods have uneven overall blackness, exhibiting extremely high blackness in the central area and high grayness in the side areas. Although they can fully display the edges of the entire workpiece, the edge details are unclear, the contrast is poor, the ability to display defects is low, and the overall visual experience is unsatisfactory. In contrast, the image obtained by the CNN-Transformer fusion method proposed in this embodiment has uniform overall blackness, rich micro-detail information in complex areas of the workpiece, moderate overall contrast, detailed edge rendering, high clarity, and the best visual experience.

[0076] To objectively evaluate the specific fusion effects of different algorithms, the fusion effects of 18 algorithms were objectively evaluated from different aspects using 8 of the objective evaluation indicators mentioned above for the same workpiece image. The specific evaluation indicators include peak signal-to-noise ratio (PSNR), structural similarity (SSIM, MS-SSIM), normalized root mean square error (NRMSE), average gradient (AG), information entropy (EN), no-reference structural similarity (NRSS), and spatial frequency (SF). The evaluation indicators of each algorithm obtained by using different algorithms for the same workpiece image are shown in Tables 2 and 3 below.

[0077] Table 2 Comparison of Objective Evaluation Indicators for Fuded Images

[0078]

[0079] A higher Peak Signal-to-Noise Ratio (PSNR) value indicates smaller pixel errors and less image distortion. Comparing the 17 image fusion methods in Table 2, it can be seen that the CNN-Transformer fusion method proposed in this embodiment produces the highest PSNR value, indicating that the fused image obtained using this method has the smallest pixel error and the least image distortion. This is consistent with the subjective evaluation of the fused image performance corresponding to this algorithm.

[0080] The larger the structural similarity (SSIM, MS-SSIM) index, the smaller the structural difference between the two images. Therefore, we can see from Table 2 that the CNN-Transformer fusion method proposed in this embodiment performs best in preserving image structural information, which is consistent with the performance of the fused image corresponding to the algorithm in this chapter in terms of structural information in subjective evaluation.

[0081] The smaller the normalized root mean square error (NRMSE) value, the smaller the error between the fused image and the reference image. As can be seen from the table, the CNN-Transformer fusion method proposed in this embodiment produces the smallest NRMSE value, indicating that the fused image obtained by this algorithm has the smallest error between it and the reference image, and the highest degree of restoration.

[0082] Table 3 Comparison of Objective Evaluation Indicators for Fuded Images

[0083]

[0084] A higher information entropy (EN) value indicates a more dispersed gray-level distribution in the image, containing a greater amount of information and richer details. Comparing the 17 image fusion algorithms in Table 3, it can be seen that the CNN-Transformer fusion method proposed in this embodiment has the highest EN value. This indicates that the fused image obtained using this algorithm has the strongest detail display capability, which is consistent with the conclusions obtained in the subjective evaluation.

[0085] A larger average gradient (AG) value indicates sharper image edges, clearer details, and more pronounced subtle texture variations. The data in the table shows that the CNN-Transformer fusion method proposed in this embodiment has the highest AG value, indicating that the fused image obtained using this method has clear edge information and high overall image clarity, confirming the conclusion in the subjective evaluation. A higher no-reference structural similarity (NRSS) value indicates higher image clarity and structural integrity. A higher spatial frequency (SF) value indicates a clearer overall image with richer details. The data in the table shows that the NRSS and SF values ​​of the CNN-Transformer fusion method proposed in this embodiment are not the highest, indicating that the method should be optimized in terms of image clarity, structural integrity, and detail representation.

[0086] Figure 9 This figure compares the edge density values ​​of different image fusion algorithms. Edge density is typically used to measure the degree to which edge information is preserved in the fused image; a higher value usually indicates clearer image edges and better detail retention. As shown in the figure, the CNN-Transformer fusion method proposed in this embodiment has a higher edge density value than the other algorithms. This indicates that this method is superior to the other compared algorithms, and can more effectively preserve the edge and structural information in the fused image, thus obtaining a fusion result with better visual quality and clearer details.

[0087] Figure 10 This section compares the sharpness of different image fusion algorithms. A higher sharpness value indicates stronger edge contrast, greater visual clarity, and more prominent detail in the final fused image. Figure 10 It can be seen that the image obtained by using the CNN-Transformer fusion method proposed in this experiment has the highest sharpness value, indicating that the image obtained by using this method is sharper than other algorithms. The fused image is visually sharper and has richer details, which can better improve the perceptual quality of the image.

[0088] Figure 11 This section compares the contrast of different image fusion algorithms. A higher contrast value indicates a greater difference between bright and dark areas in the image, resulting in a more distinct overall image hierarchy and stronger visual impact. Figure 11 As can be seen, the image obtained using the CNN-Transformer fusion method proposed in this embodiment has significantly better contrast than other fusion algorithms. This indicates that the fused image obtained using this method has high contrast, clear layers, and a better visual effect.

[0089] Based on all the evaluation metrics above, it can be concluded that the fused image quality obtained using the CNN-Transformer fusion method proposed in this embodiment achieves optimal quality in all aspects, producing high-quality fused images. While its computational cost is relatively high, it remains within an acceptable range. The CNN-Transformer fusion method significantly improves the overall quality of fused images, greatly enhancing the detection efficiency and defect detection rate of nondestructive testing (NDT) technology, and has broad application prospects and value in the future NDT industry. A comparison of the overall quality of different image fusion algorithms is shown below. Figure 12 As shown.

[0090] Table 4 shows the changes in computation time, PSNR, and SSIM with the number of sampling steps. As can be seen from Table 4, computation time increases approximately linearly with the number of sampling steps. PSNR and SSIM increase with the number of sampling steps, but there is a saturation point. In the range of 1-50 steps, PSNR and SSIM increase significantly with the number of sampling steps, with PSNR increasing from 9.6913 to 34.8312 and SSIM increasing from 0.0259 to 0.9611. In the range of 50-2000 steps, PSNR and SSIM increase slightly with the number of sampling steps, with PSNR increasing from 34.8312 to 35.5620 and SSIM increasing from 0.9611 to 0.9635.

[0091] Considering these three metrics, we can conclude that a larger number of sampling steps is not necessarily better. We need to find a balance between computation time and image quality to achieve the best image quality within an acceptable computation time.

[0092] Table 4. Changes in computation time, PSNR, and SSIM with the number of sampling steps.

[0093]

[0094] Figure 13 The results of the calculation show that the time and PSNR value change with the number of sampling steps are consistent with the analysis in Table 4 above.

[0095] Table 5 shows the impact of the number of modules on computation time, PSNR, and SSIM. The table shows that computation time increases approximately linearly with the number of modules. PSNR and SSIM also increase, but the magnitude of the increase differs. Increasing from 2 to 3 modules significantly increases PSNR by approximately 2 dB and SSIM by approximately 0.035. However, increasing from 4 to 5 modules only increases PSNR by approximately 0.3 dB and SSIM by approximately 0.004, showing a significantly reduced increase. Therefore, it can be concluded that increasing the number of modules improves the quality of the fused image, but correspondingly, the computation time of the algorithm also increases. Therefore, when setting the number of modules, computational efficiency and image quality should be considered comprehensively to maximize detection benefits.

[0096] Table 5. Changes in computation time, PSNR, and SSIM with the number of modules

[0097]

[0098] Figure 14 and Figure 15 This shows how the optimal PSNR and optimal SSIM change with each training epoch during the model training process. Figure 14 and Figure 15 The images show that the best PSNR and best SSIM improve with the increase of training rounds, indicating that the quality of the fused images is also continuously improving.

[0099] The purpose of this invention is to address the problem that, due to the limited dynamic range of X-ray imaging systems, single-energy X-rays cannot adapt to varying thicknesses, often resulting in overexposure or underexposure in practical inspections. The key technical point of this invention lies in the fact that other existing image fusion algorithms suffer from various shortcomings such as low contrast and blurred edge information. This invention overcomes these shortcomings, achieving optimal overall quality in the final fused image. This invention significantly improves the quality of the fused image, enhancing defect detection rate and inspection efficiency in practical applications, and promoting the development and advancement of nondestructive testing technology.

Claims

1. A CNN-Transformer multi-energy spectral digital ray image fusion method, characterized in that, Includes the following steps: (1) The system receives DR images obtained under different energies, and the preprocessing module performs preprocessing on the input images; (2) Input the preprocessed image into the dual-branch feature encoder: The Convolutional Neural Network (CNN) module is responsible for extracting local features; the Transformer module is responsible for extracting global features; and the feature extraction module extracts deep features from the preprocessed image. (3) The image is enhanced and fused. The enhancement block module enhances the extracted features; the cross-attention fusion decoder module uses the cross-attention mechanism to fuse the features of CNN and Transformer. (4) The image is decoded and reconstructed. The fused features are input into the decoder module and the feature representation is further optimized through the adaptive attention fusion mechanism. At the same time, the image details are restored by upsampling through multiple ConvBlock layers. (5) Use a composite loss function for training.

2. The CNN-Transformer multi-spectral digital ray image fusion method according to claim 1, characterized in that, In step (1), the preprocessing module adjusts the input image to 256×256 pixels to ensure that the size of the input image remains consistent; at the same time, it normalizes the pixel values ​​to the range of [0, 1].

3. The CNN-Transformer multi-spectral digital ray image fusion method according to claim 1, characterized in that, In step (2), the two modules of the feature extraction module work together to extract rich and complementary deep feature representations from the preprocessed image; feature extraction is performed simultaneously through two parallel paths. The convolutional neural network module structure consists of four cascaded ConvBlocks; it focuses on local features of the image and can perform layer-by-layer scanning of local details in the obtained image to extract texture, edge, and local contrast detail features. The Transformer module focuses on global features of the image. Its structure includes multi-head self-attention (MSA) and multilayer perceptron (MLP), and it uses layer normalization to stabilize training. It captures long-distance dependencies and global contextual information of the image and establishes connections between any two locations in the image through a self-attention mechanism.

4. The CNN-Transformer multi-spectral digital ray image fusion method according to claim 1, characterized in that, In step (5): Total loss ; in, Representing structural similarity loss, it minimizes the structural differences between the fused image and the target image; This represents feature-perceptual loss, which can improve the realism of the final image; L1 loss, also known as mean absolute error (MAE), enables the fused image to generate sharper edges and is less prone to over-smoothing. The three hyperparameters α, β, and γ are used to balance the weights of each loss and control their contribution to the overall optimization.

Images