Thermal image classification method based on multi-scale attention fusion

Abstract

The application discloses a thermal image classification method based on multi-scale attention fusion. After the thermal image is extracted by a shared encoder, the method inputs three parallel branches of global context, local texture and edge features; based on Dirichlet evidence theory, the evidence strength of each branch is calculated and adaptive fusion weight is generated to obtain the spatial domain fusion feature; the original thermal image is subjected to a learnable wavelet transform to extract the frequency domain feature, which is spliced with the spatial domain fusion feature and then output by a classification head. In the training stage, the classification loss, topology preservation loss and explanation consistency loss are combined for optimization. The application can adaptively adjust the fusion weight, realize complementary fusion of spatial and frequency domains, and improve the classification accuracy and explainability of the thermal image.

Description

Technical Field

[0001] This invention belongs to the field of intelligent medical image analysis technology, specifically relating to a thermal image classification method based on multi-scale attention fusion. Background Technology

[0002] Infrared thermography is a functional imaging technique that collects information based on the temperature distribution of a target surface. Due to its non-invasive, radiation-free, low-cost, and repeatable characteristics, it has been widely used in various fields such as industrial inspection, building thermal engineering, and biomedicine. In biological tissue analysis, thermography can indirectly reflect changes in metabolic activity and blood perfusion within tissues through skin surface temperature distribution, providing rich thermal characteristic information for tissue classification. Compared with traditional X-ray radiography, ultrasound, and magnetic resonance imaging (MRI), thermography does not involve ionizing radiation and has lower requirements for operating conditions, attracting continuous attention from researchers in recent years for the intelligent analysis of image data.

[0003] However, thermal images face multiple challenges in practical classification applications. First, some regions of interest often exhibit small volume, low contrast, and blurred boundaries, making them difficult to identify effectively using conventional visual observation and simple thresholding methods. Second, different models of thermal imagers vary in dynamic range, resolution, and noise levels; images of the same target acquired on different devices may show significant brightness and contrast shifts, resulting in insufficient cross-device generalization ability for algorithms relying on fixed features. Furthermore, in practical applications, the distribution of samples across different categories is often unbalanced, further increasing the difficulty for classifiers to identify the minority class.

[0004] Existing intelligent thermal image analysis methods can be broadly categorized into three types. The first type is based on traditional machine learning, typically involving manual extraction of temperature statistics, grayscale texture features, or morphological descriptors, followed by classification using classifiers such as support vector machines and random forests. These methods have low data requirements and computational overhead, but limited feature representation capabilities and weak robustness to changes in imaging conditions and noise interference. The second type is end-to-end deep learning-based solutions, represented by convolutional neural networks such as ResNet and DenseNet. These can automatically learn discriminative features directly from raw images, and their classification performance is generally superior to manual feature methods. However, single network structures still suffer from insufficient sensitivity when dealing with small target areas and low-contrast samples, and they fail to simultaneously consider both the overall morphological features and microscopic texture details of the target area. The third type is multi-branch or multi-modal fusion schemes, attempting to integrate information from different levels through early, mid, or late feature fusion strategies. However, most schemes use fixed ratios or empirical rules for fusion, lacking a mechanism for adaptive adjustment based on sample characteristics and the reliability of each branch, resulting in unstable fusion performance when facing difficult samples.

[0005] At the feature representation level, existing solutions primarily focus on extracting spatial domain information, with insufficient utilization of frequency domain features. Frequency domain decomposition of images can separate low-frequency structural information from high-frequency detail information, providing classifiers with discriminative cues complementary to spatial domain features. However, a mature solution for effectively fusing frequency domain features with the spatial domain features of deep networks within a unified framework has yet to be developed. Furthermore, regarding model interpretability, existing methods lack explicit constraints on edge feature quality and standardized guidance to ensure consistency between the model's focus areas and prior knowledge from domain experts during training. This results in poor stability of the output edge saliency map and opaque model decision-making processes, making it difficult to meet the practical interpretability requirements of high-reliability application scenarios. Summary of the Invention

[0006] This invention proposes a thermal image classification method based on multi-scale attention fusion to address the technical problems of existing thermal image classification methods, such as insufficient representation ability of difficult samples, lack of adaptability of multi-branch fusion, insufficient utilization of frequency domain information, and poor model interpretability.

[0007] To address the aforementioned technical problems, this invention provides a thermal image classification method based on multi-scale attention fusion, comprising the following steps: Step S1: Input the original thermal image into the shared encoder and extract the shared feature map through multi-level downsampling; Step S2: Input the shared feature map into the global context branch, the local texture branch, and the edge feature branch simultaneously, and extract the global context features, local texture features, and edge features respectively; Step S3: Based on Dirichlet evidence theory, calculate the evidence strength for the global context features, the local texture features, and the edge features respectively, and generate adaptive fusion weights. Use the adaptive fusion weights to perform weighted fusion of the global context features, the local texture features, and the edge features to obtain spatial fusion features. Step S4: Perform a learnable discrete wavelet transform on the original thermal image to obtain frequency domain features, and then concatenate and fuse the frequency domain features with the spatial domain fusion features to obtain joint features; Step S5: Perform global average pooling and fully connected layer classification on the joint features in sequence, and output the thermal image classification result.

[0008] Preferably, the shared encoder uses a convolutional neural network backbone network, which downsamples the resolution of the original thermal image step by step through multiple convolution operations with a stride of 2. After each downsampling, batch normalization and activation function processing are performed, and the number of channels of the shared feature map is a preset basic number of channels.

[0009] Preferably, the global context branch adopts a dilated convolution stacked structure, including: extracting features from the shared feature map through depthwise separable convolution, and then stacking multiple dilated convolutions with increasing dilation rates; A combination of channel attention and spatial attention mechanisms is applied to the outputs of multiple dilated convolutions. The channel attention mechanism calculates channel weights through global average pooling and fully connected layers, while the spatial attention mechanism calculates spatial weights through cross-channel average pooling and max pooling combined with convolutions, outputting the global context features.

[0010] Preferably, the local texture branch adopts a multi-scale parallel convolution structure, including: simultaneously inputting the shared feature map into multiple parallel depth-separable convolution branches with different kernel sizes, and extracting local texture information at different scales respectively; The outputs of multiple parallel depthwise separable convolution branches are concatenated along the channel dimension to form multi-scale texture features. The multi-scale texture features are then refined and fused using depthwise separable convolution, and enhanced by a combination of channel attention and spatial attention mechanisms to output the local texture features.

[0011] Preferably, the edge feature branch includes: initializing the convolutional layer weights using a Sobel operator, which includes a horizontal gradient template and a vertical gradient template, and calculating the edge response on the shared feature map; performing convolution processing on the edge response using depthwise separable convolution; and then enhancing the response by a combination of channel attention and spatial attention mechanisms to highlight the response of the edge region, thereby obtaining the edge features.

[0012] Preferably, in step S3, the method for calculating the strength of evidence and generating adaptive fusion weights based on Dirichlet evidence theory includes the following sub-steps: Step 1: Perform global average pooling on the global context features, the local texture features, and the edge features respectively, and then map them into evidence values ​​through a two-layer fully connected network. The first layer uses the SiLU activation function, and the second layer uses the Softplus activation function to ensure that the evidence value is non-negative. The values ​​1, 2, and 3 correspond to the global context branch, the local texture branch, and the edge feature branch, respectively. Step 2: Set the evidence value Adding 1 gives the Dirichlet distribution parameters. Calculate the basic weights Calculate the overall uncertainty where K is the number of branches; Step 3: Calculate the adjusted adaptive fusion weights The global context feature, the local texture feature, and the edge feature are multiplied by their respective adaptive fusion weights and then summed element by element. The result is then processed by depthwise separable convolution to obtain the spatial fusion feature.

[0013] Preferably, the learnable discrete wavelet transform in step S4 includes: defining four 2×2 convolution kernels, which are initialized with the coefficients of the low-frequency approximation component, horizontal high-frequency component, vertical high-frequency component, and diagonal high-frequency component of the Haar wavelet, respectively. The kernel weights for the low-frequency approximation component are initialized to [[0.5,0.5],[0.5,0.5]], the kernel weights for the horizontal high-frequency component are initialized to [[0.5,-0.5],[0.5,-0.5]], the kernel weights for the vertical high-frequency component are initialized to [[0.5,0.5],[-0.5,-0.5]], and the kernel weights for the diagonal high-frequency component are initialized to [[0.5,-0.5],[-0.5,0.5]]; the original thermal image is decomposed into four frequency bands through a convolution operation with a stride of 2; and the weights of the four convolution kernels participate in gradient updates during training.

[0014] Preferably, the concatenation and fusion of the frequency domain features and the spatial domain fusion features includes: extracting features from the four frequency bands through convolutional layers, resampling the extracted features of the four frequency bands to the same spatial resolution as the spatial domain fusion features, and concatenating them in the channel dimension to form a complete frequency domain feature representation; concatenating the spatial domain fusion features and the frequency domain feature representation in the channel dimension, compressing the channels through depthwise separable convolution, and enhancing them by applying a combination of channel attention and spatial attention mechanisms to obtain the joint features.

[0015] Preferably, during the training phase, the edge features are processed using a 1×1 convolution and a sigmoid activation function to generate an edge saliency map; the model parameters are optimized using a total loss function, wherein the total loss function... for: ; in For classification loss, the cross-entropy loss function is used to calculate the difference between the classification prediction result and the true label; The topology-preserving loss is calculated based on the skeleton structure and distance field representation of the edge saliency map; To explain the consistency loss, the joint features are averaged across the channel dimension to obtain the predicted saliency map, and the KL divergence between the predicted saliency map and the prior saliency map is calculated. and These are preset weighting coefficients.

[0016] Preferably, the topology preservation loss The calculation includes: performing soft skeletonization on the edge saliency map through multiple average pooling and ReLU activation functions to extract the skeleton structure of the edges; calculating the soft distance transform through multiple average pooling to obtain the distance field representation of the edges; calculating the skeleton structure and the distance field representation for the predicted edge saliency map and the reference edge saliency map respectively; calculating the L1 loss of the skeleton structure and the L1 loss of the distance field representation between the prediction and the reference, and obtaining the topology preservation loss by weighted summation with preset weights; The explanation consistency loss The calculation includes: averaging the joint features along the channel dimension to obtain a single-channel predicted saliency map; normalizing the predicted saliency map and the prior saliency map to make them effective probability distributions; and calculating the KL divergence between the normalized predicted saliency map and the normalized prior saliency map as the interpretation consistency loss.

[0017] The beneficial effects of the present invention include at least the following: (1) This invention achieves the synchronous extraction of multi-dimensional features of the target region through the collaborative design of three parallel paths: shared encoder, global context branch, local texture branch, and edge feature branch. The three branches focus on different scales and types of information, which are highly complementary and effectively improve the representation ability of small targets and low contrast samples.

[0018] (2) The uncertainty perception fusion mechanism constructed based on Dirichlet evidence theory can automatically evaluate the credibility of each branch on the current sample and allocate fusion weight accordingly. In high uncertainty scenarios, the weights tend to be balanced to reduce the negative impact of erroneous branches, and in low uncertainty scenarios, the contribution of the advantageous branches is amplified to improve the discrimination accuracy.

[0019] (3) This invention introduces a learnable discrete wavelet transform initialized with Haar wavelets, which decomposes the original image into low-frequency structural components and high-frequency detail components. After feature extraction and scale alignment, the components are stitched together with spatial fusion features, realizing the complementary fusion of spatial and frequency domain information and enhancing the model's ability to perceive texture and edge patterns.

[0020] (4) In the training phase, the present invention uses topology preservation loss and interpretation consistency loss together. The former ensures the connectivity and structural stability of the edge saliency map through soft skeletonization operation and distance field constraint, while the latter aligns the model’s attention distribution with the prior attention region labeled by the domain expert through KL divergence, making the model’s decision basis more transparent and more consistent with the expert’s prior knowledge, thus meeting the actual needs of interpretability in high reliability application scenarios. Attached Figure Description

[0021] Figure 1This is a schematic diagram of the overall architecture flow of an embodiment of the present invention; Figure 2 This is a schematic diagram of a local high-temperature anomaly in a thermal image according to an embodiment of the present invention; Figure 3 This is a schematic diagram of abnormal blood vessels in a thermographic image according to an embodiment of the present invention. Detailed Implementation

[0022] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the protection scope of the present invention.

[0023] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0024] This embodiment uses binary classification of breast thermograms as an application scenario. The dataset used contains 1200 thermal images acquired by an infrared thermal imager under a standardized acquisition process, including 780 samples of category 1 (benign) and 420 samples of category 2 (malignant). The dataset is randomly divided into training, validation, and test sets in a ratio of 7:1.5:1.5, ensuring that multiple records of the same subject are not distributed across sets. All input images are uniformly scaled to 480×640 pixels, and pixel values ​​are normalized to the [0,1] interval.

[0025] The overall processing flow of the method of the present invention is as follows: Figure 1 As shown, the system consists of a shared encoder module, a spatial domain three-branch parallel feature extraction module (including a global context branch submodule, a local texture branch submodule, and an edge feature branch submodule), an uncertainty-aware hierarchical adaptive fusion module (including a branch evidence extraction submodule, an uncertainty gating weight calculation submodule, and a weighted feature fusion submodule), a frequency domain-spatial domain dual-path fusion module (including a learnable discrete wavelet transform submodule, a frequency domain feature extraction and scale alignment submodule, and a frequency domain-spatial domain fusion submodule), a post-processing and classification decision module, and a training and loss calculation module. After the original thermal image is processed by the shared encoder to extract multi-scale semantic features, the three parallel branches extract features from the global context, local texture, and edge boundaries, respectively. The three sets of extracted features are adaptively weighted and fused through an uncertainty-aware fusion mechanism to obtain spatial domain fusion features. Simultaneously, the original image is processed by a learnable discrete wavelet transform to generate frequency domain features, which are then concatenated and fused with the spatial domain fusion features to obtain joint features. Finally, the classification decision module outputs the classification result. The specific implementation of each module is explained step-by-step in the order of the processing flow below.

[0026] Step S1: Input the original thermal image into the shared encoder and extract the shared feature map through multi-level downsampling.

[0027] The preprocessed raw thermal image is input into the shared encoder module, which employs a replaceable convolutional neural network backbone or a lightweight pyramid encoder. In this embodiment, the shared encoder uses ResNet-34 as the backbone network, comprising five levels of downsampling. Each level halves the spatial resolution of the feature map through a convolution operation with a stride of 2. After five levels of downsampling, the feature map resolution is reduced to 1 / 32 of the original input. After each downsampling level, batch normalization and SiLU activation function processing are performed sequentially. Batch normalization stabilizes the numerical distribution of features in each layer by normalizing the mean and variance of the feature values ​​of a small batch of samples, ensuring the stability of the feature distribution and the smoothness of gradient flow. The final output channel number of the encoder is mapped to a preset base channel number, and this feature map serves as the unified input for the subsequent three parallel branches. In actual deployment, the backbone network can be replaced with lightweight architectures such as EfficientNet-B0 or ​​MobileNet-V3. When replacing, only the spatial size and channel number of the output feature map need to be kept consistent with the interface of the subsequent modules.

[0028] Step S2: Input the shared feature map into the global context branch, the local texture branch, and the edge feature branch simultaneously, and extract the global context features, local texture features, and edge features respectively.

[0029] Specifically, the three branches each have their own focus in terms of architecture design, and are respectively dedicated to extracting the overall morphological information, multi-scale texture details and edge boundary structure of the target area.

[0030] (a) Global Context Branch In this embodiment of the invention, the global context branch employs a stacked dilated convolution structure to acquire a wide range of spatial context information. Specifically, the shared feature map first undergoes preliminary feature extraction and channel mapping through a depthwise separable convolution layer. Subsequently, the feature map sequentially passes through three dilated convolution layers with increasing dilation rates. In this embodiment, the dilation rates of the three dilated convolutions are set to 1, 2, and 4, respectively, and the padding values ​​are set to values ​​equal to the dilation rates to ensure that the spatial resolution of the feature map remains consistent before and after convolution. A dilated convolution with a dilation rate of 1 is equivalent to a standard convolution, focusing on the local relationships between adjacent pixels; a dilated convolution with a dilation rate of 2 samples at one-pixel intervals, expanding the effective receptive field to a 5×5 range; and a dilated convolution with a dilation rate of 4 further expands the effective receptive field to a 9×9 range. After stacking the three dilated convolution layers, the receptive field is expanded while maintaining the spatial resolution of the feature map, enabling feature points to perceive context information over a larger range.

[0031] Channel attention and spatial attention mechanisms are applied to the output of the three-layer dilated convolution. Channel weights are calculated through global average pooling and fully connected layers, while spatial weights are calculated through a combination of cross-channel average pooling and max pooling convolutions, enabling adaptive adjustment of response intensity for different spatial regions. This adaptively enhances the response of important feature channels and spatial locations. When the lesion region is large, the boundary is blurred, or global morphological information is required, this branch can provide effective global contextual representation, which helps improve classification accuracy.

[0032] (ii) Local texture branches Local texture branches employ multi-scale parallel convolution structures to capture texture details at different granularities. For example... Figure 2 As shown, local high-temperature areas in thermal images often exhibit different texture patterns and temperature gradient distributions compared to surrounding tissues. These local texture differences are important criteria for classification. The shared feature map is simultaneously fed into three parallel depthwise separable convolutional branches. In this embodiment, the kernel sizes of the three branches are 3×3, 5×5, and 7×7, respectively. The 3×3 convolutional kernel focuses on capturing fine local texture patterns, such as subtle temperature gradient changes and fine-grained textures; the 5×5 convolutional kernel covers a medium range of texture structures, suitable for detecting medium-scale patch or stripe patterns; and the 7×7 convolutional kernel has a strong response capability to larger-scale texture changes. Each branch uses depthwise separable convolution to control the parameter scale and reduce computational cost. After each branch independently extracts texture information at different scales, the output feature maps of the three branches are concatenated along the channel dimension to form a composite feature representation containing multi-scale texture information.

[0033] The stitched multi-scale texture features are refined and fused through an additional depthwise separable convolution layer. This compresses the increased channel count caused by stitching back to the same number of channels as the input, while simultaneously promoting the interaction and integration of information between branches of different scales. The refined features are then processed sequentially through channel attention and spatial attention mechanisms to enhance the response of important texture features. Finally, this branch outputs a texture feature representation with the same spatial resolution as the input feature map. When faced with subtle texture differences between different categories, this branch can provide richer discriminative cues and more effective local detail representations than single-scale convolution, which helps to distinguish the input thermal image; in this embodiment, it distinguishes subtle texture differences between benign and malignant lesions.

[0034] (iii) Edge feature branches In this embodiment of the invention, the edge feature branch employs a strategy combining Sobel operator initialization and learnable enhancement to extract boundary information and vascular structures of the target region. For example... Figure 3As shown, abnormal vascular structures in thermal images exhibit distinct edge features in their thermal distribution. This edge information is valuable for distinguishing different types of thermal patterns. Specifically, the weights of a standard convolutional layer are initialized using the gradient templates of the Sobel operator. The horizontal gradient template is set to [[-1,0,1],[-2,0,2],[-1,0,1]], and the vertical gradient template is set to [[-1,-2,-1],[0,0,0],[1,2,1]]. This initialization method allows the convolutional layer to possess the prior ability of classical edge detection from the early stages of training, enabling it to extract gradient responses in the horizontal and vertical directions from the feature map. Unlike a fixed Sobel operator, the weights of this convolutional layer participate in gradient updates during subsequent training, thus utilizing the prior knowledge of classical edge detection while maintaining learnability.

[0035] Sobel initializes the edge responses of the convolutional layer outputs and then performs further feature processing through depthwise separable convolutions, making the edge features clearer and suppressing false edges caused by noise, thus better adapting to the specific distribution of thermal image data. The processed features are also further processed by channel attention and spatial attention mechanisms. This branch ultimately outputs edge features. During the training phase, an additional 1×1 convolutional layer and a sigmoid activation function generate an edge saliency map with values ​​between [0,1]. This edge saliency map is used to calculate the subsequent topology-preserving loss. For target regions with clear boundaries, anomalous shapes, or significant changes in vascular structure, the edge feature branch can provide effective edge feature representations, offering intuitive and interpretable discriminative information for classification.

[0036] Step S3: Based on Dirichlet evidence theory, calculate the evidence strength for global context features, local texture features and edge features respectively and generate adaptive fusion weights. Use the adaptive fusion weights to perform weighted fusion of global context features, local texture features and edge features to obtain spatial fusion features.

[0037] The features output by each of the three branches vary in terms of information content and reliability depending on the sample. For some samples, global morphological features may be the most discriminative clues; while for others, local texture or edge information may be more crucial. A fixed-ratio fusion strategy cannot adapt to such differences between samples. Therefore, this invention introduces an uncertainty-aware fusion mechanism based on Dirichlet evidence theory, enabling the fusion weights to be automatically adjusted according to the credibility of each branch on the current sample.

[0038] First, global average pooling is performed on the global context features, local texture features, and edge features respectively, compressing the two-dimensional feature maps of each branch into one-dimensional global feature vectors. Then, the global feature vector of each branch is mapped through an evidence extraction module consisting of two fully connected layers: the first fully connected layer uses the SiLU activation function for non-linear transformation, and the second fully connected layer uses the Softplus activation function to ensure the output is non-negative; this non-negative output is the evidence value for the current branch. The evidence values ​​for the three branches are denoted as follows: , and The subscripts 1, 2, and 3 correspond to the global context branch, the local texture branch, and the edge feature branch, respectively. When the branch features are clear, information-rich, or highly relevant to the task, the evidence value increases accordingly; when the branch features are blurry, noisy, or have low relevance to the task, the evidence value decreases accordingly.

[0039] Then, the evidence value of each branch is incremented by 1 to obtain the parameters of the Dirichlet distribution. : ; Calculate the basic weights of each branch based on Dirichlet distribution parameters. : ; Simultaneously calculate the overall uncertainty measure. : ; Where K is the number of branches, and in this embodiment, K is 3. When the total evidence for the three branches is small, uncertainty increases. The value is relatively large; when the total evidence is sufficient. Approaching zero.

[0040] Based on the uncertainty, the basic weights are adjusted to obtain the adaptive fusion weights: ; When uncertainty is high ( (larger) It tends to distribute more evenly, thus reducing over-reliance on a single branch; when uncertainty is low ( (smaller) Closer to the base weights This allows the dominant branches to play a larger role in the integration process.

[0041] After obtaining the three sets of adaptive fusion weights, the feature map of each branch is multiplied element-wise by its corresponding weight. Then, the three weighted feature maps are summed element-wise. Finally, a depthwise separable convolution layer is used for feature refinement and channel integration to eliminate potential redundancy and inconsistency between features from different branches, outputting spatial fusion features. This fusion strategy can adaptively highlight the contribution of important branches according to sample characteristics, thereby improving classification performance.

[0042] Step S4: Perform learnable discrete wavelet transform on the original thermal image to obtain frequency domain features, and then stitch and fuse the frequency domain features with the spatial domain fusion features to obtain joint features.

[0043] The spatial fusion features mainly contain discriminative information extracted from the deep semantic level. In order to further enhance the model's ability to perceive texture details and periodic patterns, this invention introduces a frequency domain analysis path to complement and fuse the frequency domain features of the original thermal image with the spatial fusion features.

[0044] The frequency domain path employs learnable discrete wavelet transform (DWT) to perform frequency band decomposition on the original thermal image. Specifically, four 2×2 convolution kernels are defined and initialized with the coefficients of the four components of the standard Haar wavelet: the convolution kernel for the low-frequency approximation component (LL) is initialized to [[0.5,0.5],[0.5,0.5]], the convolution kernel for the horizontal high-frequency component (LH) is initialized to [[0.5,-0.5],[0.5,-0.5]], the convolution kernel for the vertical high-frequency component (HL) is initialized to [[0.5,0.5],[-0.5,-0.5]], and the convolution kernel for the diagonal high-frequency component (HH) is initialized to [[0.5,-0.5],[-0.5,0.5]]. Four convolutional kernels perform convolution operations on the input image with a stride of 2, decomposing the input image into four frequency bands with half the spatial resolution. The LL component preserves the overall structure and low-frequency energy distribution of the image, the LH component reflects horizontal edge and texture variations, the HL component reflects vertical edge and texture variations, and the HH component captures high-frequency details in the diagonal direction. The weights of the four convolutional kernels participate in gradient updates during training, enabling the wavelet basis function to gradually adapt to the frequency domain feature distribution of the thermal image, rather than adhering to the standard Haar wavelet shape.

[0045] The four frequency bands obtained from the decomposition are each processed through their respective convolutional layers for feature extraction. The number of channels in each frequency band is mapped to a preset number of output channels, such as half the number of basic channels. Since wavelet decomposition uses a convolution operation with a stride of 2, the spatial resolution of each frequency band is reduced to half of the original input. Furthermore, the spatial resolution of the spatial domain fusion features, after multiple downsampling stages by the shared encoder, is much smaller than that of the frequency band features. Therefore, it is necessary to resample the four frequency band features to the same spatial resolution as the spatial domain fusion features using bilinear interpolation. The upsampled four frequency band features are then concatenated along the channel dimension to form a complete frequency domain feature representation.

[0046] Spatial-domain fusion features and frequency-domain feature representations are concatenated along the channel dimension to form a spatial-frequency joint feature. Then, depthwise separable convolution is used for channel compression, restoring the joint feature's channel count to the same preset base channel count as the spatial-domain fusion feature, thus achieving the fusion and channel alignment of spatial and frequency domain information. The compressed features are then passed through channel attention and spatial attention mechanisms to enhance the response of important features, outputting the final joint feature. This fusion strategy fully utilizes the complementary feature representations of the spatial and frequency domains, thereby improving the model's representational ability and classification performance.

[0047] In another embodiment of the present invention, after the frequency-spatial domain fusion outputs the joint features and before they are fed into the classification head, an optional post-processing attention submodule is provided. When this submodule is enabled, the same channel attention mechanism and spatial attention mechanism as described above are applied to the joint features again, performing secondary refinement on the fused features to eliminate redundant information and residual noise that may be introduced during the frequency-spatial domain fusion process, making the feature representation more compact and more discriminative. When this submodule is not enabled, the joint features output by the frequency-spatial domain fusion module are directly fed into the subsequent classification head.

[0048] Step S5: Perform global average pooling and fully connected layer classification on the joint features in sequence, and output the thermal image classification results.

[0049] Global average pooling is performed on the joint features to compress the spatial dimension to 1×1, resulting in a global feature vector with channel dimensions. This vector is flattened and fed into a classification head consisting of two fully connected layers. The first fully connected layer of the classification head maps the feature dimension to an intermediate dimension; in this embodiment, the larger of half the base number of channels and 32 is chosen. The SiLU activation function and Dropout regularization with a dropout rate of 0.2 are used to alleviate overfitting. The second fully connected layer maps the feature dimension to the number of classes; in this embodiment, it is 2, outputting unnormalized classification logits. Finally, the logits are converted into a class probability distribution using the Softmax function, and the class with the higher probability is taken as the final classification result.

[0050] In practical applications, the optimal threshold can be determined by plotting the Receiver Operating Characteristic (ROC) curve on the validation set and selecting the point corresponding to the maximum Youden Index, based on the trade-offs required for different category recognition rates.

[0051] In this embodiment of the invention, the total loss function It consists of a weighted combination of three parts: ; in For classification loss, the standard cross-entropy loss function is used to calculate the difference between the classification logits and the true label. This loss function can effectively punish misclassification and guide the model to learn the correct benign and malignant discrimination rules. During training, this loss serves as the main optimization objective, driving the update of model parameters.

[0052] The calculation process for the topology-preserving loss is as follows: A soft skeletonization operation is performed on the edge saliency map output by the edge feature branch. This involves extracting the skeleton structure of the edges through multiple consecutive average pooling and ReLU activation functions to obtain the skeleton structure representation of the edges. Simultaneously, a soft distance transform is calculated through multiple consecutive average pooling operations to obtain the distance field representation of the edges. The skeleton structure and distance field are calculated for both the edge saliency map predicted by the model and the reference edge saliency map automatically generated based on the ground truth labels. Then, the L1 loss of the skeleton structure and the L1 loss of the distance field between the prediction and the reference are calculated, and the topology-preserving loss is obtained by weighting and summing them using preset weights. This loss function can constrain the edge saliency map to maintain connectivity and skeleton stability, thereby improving the quality and interpretability of edge features.

[0053] To explain the consistency loss, the calculation process is as follows: The joint features are averaged across the channel dimension to obtain a single-channel predicted saliency map, which reflects the key regions the model focuses on. The predicted saliency map and the prior saliency map annotated by domain experts are normalized to ensure they each constitute an effective probability distribution. Then, the KL divergence between the normalized predicted saliency map and the prior saliency map is calculated as the explained consistency loss, measuring the difference between the predicted and prior saliency maps. This loss function can constrain the model's attention mechanism to align with physicians' diagnostic habits, thereby improving the model's interpretability and clinical applicability.

[0054] Weighting coefficient and The ranges were determined by performing a grid search on the validation set, with search ranges of {0.01, 0.05, 0.1, 0.2, 0.5} and {0.01, 0.02, 0.05, 0.1, 0.2}, and the highest validation set AUC was used as the selection criterion. In this embodiment, the search... =0.1、 The overall performance is optimal when the value is 0.05.

[0055] The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described; only preferred embodiments of the present invention are illustrated. The descriptions are relatively specific and detailed, but they should not be construed as limiting the scope of the present invention. As long as the combination of these technical features does not contradict each other, it should be considered within the scope of this specification.

[0056] It should be noted that those skilled in the art can make various modifications and improvements without departing from the inventive concept, and these all fall within the scope of protection of this invention. Therefore, the scope of protection of this invention should be determined by the appended claims.

Claims

1. A thermal image classification method based on multi-scale attention fusion, characterized in that, Includes the following steps: Step S1: Input the original thermal image into the shared encoder and extract the shared feature map through multi-level downsampling; Step S2: Input the shared feature map into the global context branch, the local texture branch, and the edge feature branch simultaneously, and extract the global context features, local texture features, and edge features respectively; Step S3: Based on Dirichlet evidence theory, calculate the evidence strength for the global context features, the local texture features, and the edge features respectively, and generate adaptive fusion weights. Use the adaptive fusion weights to perform weighted fusion of the global context features, the local texture features, and the edge features to obtain spatial fusion features. Step S4: Perform a learnable discrete wavelet transform on the original thermal image to obtain frequency domain features, and then concatenate and fuse the frequency domain features with the spatial domain fusion features to obtain joint features; Step S5: Perform global average pooling and fully connected layer classification on the joint features in sequence, and output the thermal image classification result; Step S3, which calculates the strength of evidence and generates adaptive fusion weights based on Dirichlet evidence theory, includes the following sub-steps: Step 1: Perform global average pooling on the global context features, the local texture features, and the edge features respectively, and then map them into evidence values ​​through a two-layer fully connected network. The first layer uses the SiLU activation function, and the second layer uses the Softplus activation function to ensure that the evidence value is non-negative. The values ​​1, 2, and 3 correspond to the global context branch, the local texture branch, and the edge feature branch, respectively. Step 2: Set the evidence value Adding 1 gives the Dirichlet distribution parameters. Calculate the basic weights Calculate the overall uncertainty where K is the number of branches; Step 3: Calculate the adjusted adaptive fusion weights The global context feature, the local texture feature, and the edge feature are multiplied by their respective adaptive fusion weights and then summed element by element. The result is then processed by depthwise separable convolution to obtain the spatial fusion feature.

2. The method according to claim 1, characterized in that, The shared encoder uses a convolutional neural network backbone network, which downsamples the resolution of the original thermal image step by step through multiple convolution operations with a stride of 2. After each downsampling, batch normalization and activation function processing are performed. The number of channels in the shared feature map is a preset basic number of channels.

3. The method according to claim 1, characterized in that, The global context branch adopts a dilated convolution stacked structure, including: extracting features from the shared feature map through depthwise separable convolution, and then stacking multiple dilated convolutions with increasing dilation rates. A combination of channel attention and spatial attention mechanisms is applied to the outputs of multiple dilated convolutions. The channel attention mechanism calculates channel weights through global average pooling and fully connected layers, while the spatial attention mechanism calculates spatial weights through cross-channel average pooling and max pooling combined with convolutions, outputting the global context features.

4. The method according to claim 1, characterized in that, The local texture branch adopts a multi-scale parallel convolution structure, including: simultaneously inputting the shared feature map into multiple parallel depth-separable convolution branches with different kernel sizes to extract local texture information at different scales; The outputs of multiple parallel depthwise separable convolution branches are concatenated along the channel dimension to form multi-scale texture features. The multi-scale texture features are then refined and fused using depthwise separable convolution, and enhanced by a combination of channel attention and spatial attention mechanisms to output the local texture features.

5. The method according to claim 1, characterized in that, The edge feature branch includes: initializing the convolutional layer weights using the Sobel operator, which includes a horizontal gradient template and a vertical gradient template, and calculating the edge response on the shared feature map; performing convolution processing on the edge response using depthwise separable convolution; and then enhancing the response by combining channel attention and spatial attention mechanisms to highlight the response of the edge region, thereby obtaining the edge features.

6. The method according to claim 1, characterized in that, The learnable discrete wavelet transform in step S4 includes: defining four 2×2 convolution kernels, which are initialized with the coefficients of the low-frequency approximation component, horizontal high-frequency component, vertical high-frequency component, and diagonal high-frequency component of the Haar wavelet, respectively. The convolution kernel weights for the low-frequency approximation component are initialized to [[0.5,0.5],[0.5,0.5]], the convolution kernel weights for the horizontal high-frequency component are initialized to [[0.5,-0.5],[0.5,-0.5]], the convolution kernel weights for the vertical high-frequency component are initialized to [[0.5,0.5],[-0.5,-0.5]], and the convolution kernel weights for the diagonal high-frequency component are initialized to [[0.5,-0.5],[-0.5,0.5]]; decomposing the original thermal image into four frequency bands through a convolution operation with a stride of 2; and the weights of the four convolution kernels participate in gradient updates during training.

7. The method according to claim 6, characterized in that, The concatenation and fusion of the frequency domain features and the spatial domain fusion features includes: extracting features from the four frequency bands through convolutional layers, resampling the extracted features of the four frequency bands to the same spatial resolution as the spatial domain fusion features, and concatenating them in the channel dimension to form a complete frequency domain feature representation; concatenating the spatial domain fusion features and the frequency domain feature representation in the channel dimension, compressing the channels through depthwise separable convolution, and enhancing them by applying a combination of channel attention and spatial attention mechanisms to obtain the joint features.

8. The method according to claim 1, characterized in that, During the training phase, edge features are processed using 1×1 convolution and a sigmoid activation function to generate edge saliency maps; the model parameters are optimized using a total loss function. for: ； in For classification loss, the cross-entropy loss function is used to calculate the difference between the classification prediction result and the true label; The topology-preserving loss is calculated based on the skeleton structure and distance field representation of the edge saliency map; To explain the consistency loss, the joint features are averaged across the channel dimension to obtain the predicted saliency map, and the KL divergence between the predicted saliency map and the prior saliency map is calculated. and These are preset weighting coefficients.

9. The method according to claim 8, characterized in that, The topology preservation loss The calculation includes: performing soft skeletonization on the edge saliency map through multiple average pooling and ReLU activation functions to extract the skeleton structure of the edges; calculating the soft distance transform through multiple average pooling to obtain the distance field representation of the edges; calculating the skeleton structure and the distance field representation for the predicted edge saliency map and the reference edge saliency map respectively; calculating the L1 loss of the skeleton structure and the L1 loss of the distance field representation between the prediction and the reference, and obtaining the topology preservation loss by weighted summation with preset weights; The explanation consistency loss The calculation includes: averaging the joint features along the channel dimension to obtain a single-channel predicted saliency map; normalizing the predicted saliency map and the prior saliency map to make them effective probability distributions; and calculating the KL divergence between the normalized predicted saliency map and the normalized prior saliency map as the interpretation consistency loss.

Images