A multi-modal feature fusion and intelligent analysis system based on thermal tomographic images

Abstract

The application discloses a kind of based on multi-modal feature fusion and intelligent analysis system of thermal chromatography image, including preprocessing module, segmentation module, thermal chromatography module and fusion analysis module.Preprocessing module generates negative film graph with the temperature value of thermal chromatography image image and constitutes multi-modal input;With negative film graph as input, segmentation module outputs blood vessel mask, breast effective area mask, nipple mask and lesion location mask;Thermal chromatography module obtains the Q-R curve feature of lesion location;Fusion analysis module receives the above-mentioned mask and Q-R curve feature, after three trainable sub-modules of blood vessel analysis, high-temperature area asymmetry analysis, nipple asymmetry analysis and the Q-R curve analysis sub-module of gradient truncation are applied are comprehensively processed, and output benign and malignant classification results.The system uses phased training strategy and adaptive upsampling method based on sample difficulty, and provides blood vessel abnormal score, asymmetry score and Q-R curve index value as explainable analysis reference information in inference stage.

Description

Technical Field

[0001] This invention belongs to the field of intelligent medical image analysis technology, specifically relating to a multimodal feature fusion and intelligent analysis system based on thermal tomography images. Background Technology

[0002] Traditional breast imaging primarily relies on methods such as mammography, ultrasound, and MRI, but these methods have limitations including radiation exposure, high costs, and complex procedures. Thermal tomography, as a non-invasive and radiation-free method, reflects the physiological state of internal tissues by detecting the temperature distribution on the body surface, providing a new technical approach for breast health assessment. However, current thermal tomography analysis techniques still have many shortcomings.

[0003] Traditional machine learning-based thermal tomography image analysis schemes primarily employ manual feature extraction methods. These methods calculate temperature statistics, texture features, and shape features, and then use classifiers such as support vector machines and random forests to classify benign and malignant images. However, these methods rely on manual feature extraction, fail to fully utilize the multimodal characteristics of thermal tomography images, and have limited generalization capabilities.

[0004] In deep learning-based single-modal analysis schemes, convolutional neural networks, residual networks, and densely connected networks are commonly used to learn features and classify thermal tomography images. Although these methods have improved feature extraction, they still have the following shortcomings: they only use the original image as input, failing to fully utilize multimodal data such as negative images; they lack explicit modeling of medical prior knowledge such as vascular morphology features and bilateral symmetry features; the sample imbalance handling strategy is simplistic; and each functional module is trained independently, lacking a reasonable multi-module collaborative training strategy.

[0005] In recent years, some studies have begun to explore multimodal data fusion techniques, employing early, mid, or late fusion strategies, which have improved analytical results to some extent. However, the fusion strategies of such schemes are often fixed and lack adaptability; for medical prior knowledge such as vascular analysis and asymmetric analysis, they are usually used as post-processing steps in the inference stage and are not integrated into the model training process; there is a lack of effective phased training mechanisms between the segmentation module and the fusion analysis module; and there is a lack of medically interpretable design. Summary of the Invention

[0006] This invention proposes a multimodal feature fusion and intelligent analysis system based on thermal tomography images to solve the following technical problems: First, existing methods mainly rely on feature extraction and analysis of a single modality, failing to fully utilize the multimodal information of thermal tomography images for comprehensive analysis, resulting in insufficient feature utilization and limited classification accuracy.

[0007] Second, existing methods treat medical prior knowledge such as vascular analysis and asymmetric analysis only as post-processing steps in the reasoning stage, failing to integrate these important medical features into the model training process, thus limiting the model's discriminative ability and generalization performance.

[0008] Third, existing methods lack reasonable training strategies when training multi-module systems, and multi-module joint training suffers from problems such as complex gradient propagation and difficulty in fully optimizing each module.

[0009] Fourth, existing methods are not sophisticated enough in handling data imbalance and lack adaptive upsampling strategies based on the difficulty of the samples.

[0010] Fifth, existing methods lack interpretability, making it difficult for clinicians to understand the classification criteria of the models.

[0011] To address the aforementioned technical problems, this invention provides a multimodal feature fusion and intelligent analysis system based on thermal tomography images, comprising a preprocessing module, a segmentation module, a thermal tomography module, and a fusion analysis module; The preprocessing module inverts the temperature values ​​of the thermal tomography image to generate a negative image, and the thermal tomography image and the negative image constitute a multimodal input; The segmentation module takes the negative image as input and outputs a blood vessel mask, a breast effective region mask, a nipple mask, and a lesion location mask. The thermal tomography module takes the thermal tomography image and the lesion location mask as input, and obtains the QR curve features for the lesion location through the biological heat transfer equation. The fusion analysis module receives the vascular mask, the effective breast region mask, the nipple mask, and the QR curve features. After processing by the vascular analysis submodule, the high-temperature region asymmetric analysis submodule, the nipple asymmetric analysis submodule, and the QR curve analysis submodule with gradient truncation, it outputs the benign / malignant classification result.

[0012] Preferably, the preprocessing module generates the negative image by inverting the temperature value through the following steps: First, the temperature value is truncated at low temperature, and the region where the temperature value is less than the difference between the maximum temperature value and the threshold is uniformly set as the truncation value to retain the high temperature region information; then, the truncated temperature difference value is linearly mapped to the gray value range of 0 to 255; and finally, the negative image is obtained by subtracting the gray value from 255. The preprocessing module also includes an adaptive threshold adjustment mechanism, which decreases the threshold when the average gray value of the negative image is lower than a preset lower limit, and increases the threshold when the average gray value of the negative image is higher than a preset upper limit, and iteratively adjusts until the average gray value of the negative image meets the preset target range.

[0013] Preferably, the system further includes an adaptive upsampling module. The adaptive upsampling module takes the thermal tomographic images of malignant samples in the training set as input, calculates the temperature gradient amplitude distribution characteristics between adjacent pixels of each malignant sample as the temperature gradient complexity, calculates the temperature distribution variance as the thermal distribution uniformity, and weights and fuses the temperature gradient complexity and the thermal distribution uniformity to obtain a comprehensive difficulty score. Based on the comprehensive difficulty score, the malignant samples are divided into three levels: low difficulty, medium difficulty, and high difficulty. Geometric transformation upsampling is performed on low difficulty samples, combined geometric transformation and color transformation upsampling is performed on medium difficulty samples, and combined geometric transformation, color transformation, and noise addition upsampling is performed on high difficulty samples. The higher the difficulty level, the larger the upsampling factor, thereby obtaining a class-balanced training dataset.

[0014] Preferably, the formula for calculating the comprehensive difficulty score is: ; Wherein, D is the overall difficulty score, ranging from 0 to 1; G is the temperature gradient complexity, ranging from 0 to 1, obtained by normalizing the temperature gradient amplitude distribution characteristics between adjacent pixels; H is the heat distribution uniformity, ranging from 0 to 1, obtained by normalizing the temperature distribution variance. and The weighting coefficients and ;when When a sample is deemed low-difficulty, it is upsampled by 1 to 2 times and subjected to geometric transformation; when When a sample is classified as medium difficulty, it is upsampled by 3 to 4 times and combined with geometric and color transformations; when If a sample is identified as a high-difficulty sample, a combination of 5 to 6 times upsampling, geometric transformation, color transformation, and noise addition is employed.

[0015] Preferably, the segmentation module includes a first segmentation module and a second segmentation module; The first segmentation module takes the negative image as input, adopts an encoder-decoder network architecture, and simultaneously outputs the blood vessel mask, the effective breast region mask, and the nipple mask through a multi-head output design; The second segmentation module takes the cropped image of the breast region as input, segments the lesion location, and outputs the lesion location mask.

[0016] Preferably, the method for cropping the breast region includes: calculating the minimum bounding rectangle using the breast effective region mask output by the first segmentation module, cropping the negative image and then normalizing its size to obtain a fixed-size image containing only the breast effective region as input to the second segmentation module, thus forming a cascaded segmentation process of full-image coarse segmentation, breast effective region cropping, and lesion fine segmentation.

[0017] Preferably, in the fusion analysis module: the vascular analysis submodule adopts a lightweight convolutional neural network architecture, the input is the vascular mask output by the first segmentation module and the thermal tomography image, and the output is a vascular abnormality score with a value range of 0 to 1; the high-temperature region asymmetry analysis submodule adopts a lightweight convolutional neural network architecture, the input is the thermal tomography image of both breasts and the breast effective region mask output by the first segmentation module, and the output is a high-temperature region asymmetry score with a value range of 0 to 1; the nipple asymmetry analysis submodule adopts a lightweight convolutional neural network architecture, the input is the nipple mask output by the first segmentation module and the thermal tomography image of both nipple regions, and the output is a nipple asymmetry score with a value range of 0 to 1; the QR curve analysis submodule analyzes the QR curve features and outputs QR curve index values.

[0018] Preferably, the system employs a phased training strategy, comprising three phases executed sequentially: The first phase trains the first segmentation module, and the loss function is... for: ; in This indicates loss due to vessel segmentation. This indicates the loss of effective segmentation of the breast tissue. This represents the nipple region segmentation loss. After training, the network parameters of the first segmentation module are fixed. The second phase trains the second segmentation module, with the loss function being... : ; in The segmentation loss at the lesion location is used as the basis for fixing the network parameters of the second segmentation module after training. The third stage of training involves the vascular analysis submodule, the high-temperature region asymmetric analysis submodule, the nipple asymmetric analysis submodule, and the fusion layer within the fusion analysis module. The QR curve index values ​​of the QR curve analysis submodule, after gradient truncation, are used as fixed inputs, and the loss function... for: ; in Indicates loss in vascular analysis. This indicates the loss from asymmetric analysis in the high-temperature region. The loss for nipple asymmetry analysis is represented by the binary cross-entropy loss function. , and These are the balancing weighting coefficients.

[0019] Preferably, the system further includes a symmetry constraint training mechanism, which applies mirror consistency constraints to samples labeled as benign in each stage of the phased training strategy: During the training phase of the first segmentation module, an input pair of the thermal tomography image and the left and right mirror images about the human body's midline is constructed, and the mask overlap consistency loss is calculated between the segmentation output and the original segmentation output after the inverse mirror transformation. During the training phase of the second segmentation module, an input pair of the original breast region image and the left and right mirror images of the breast region is constructed, and the mirror consistency loss is applied to the lesion location segmentation result. During the training phase of the fusion analysis module, scores are extracted from the left and right breast regions respectively. A consistency constraint on the scores of the left and right sides is applied to samples labeled as benign, while no symmetry constraint is applied to samples labeled as malignant.

[0020] Preferably, both the first segmentation module and the second segmentation module use a lightweight mobile network as the encoder backbone network, and achieve feature extraction through depthwise separable convolution and inverted residual structure, and construct symmetrical upsampling layers on the backbone network to form an encoder-decoder network architecture; During the inference phase, the system outputs the benign / malignant classification result by the fusion analysis module, and the second segmentation module outputs the lesion location information through result visualization. When the benign / malignant classification result is abnormal, the system outputs an abnormality source prompt information based on whether the vascular abnormality score, the high temperature zone asymmetry score, the nipple asymmetry score, and the QR curve index value exceed the corresponding threshold.

[0021] The beneficial effects of the present invention include at least the following: (1) The present invention uses a preprocessing module to invert the temperature value of the thermal tomography image to generate a negative image, so that the original thermal tomography image and the negative image form a complementary multimodal input. While retaining key thermal information, it improves image contrast and feature diversity, and improves the accuracy of subsequent segmentation and fusion analysis.

[0022] (2) The present invention adopts a phased training strategy, which divides the system training into three stages: the first segmentation module training, the second segmentation module training, and the fusion analysis module training. After each stage is completed, the network parameters of the module in that stage are fixed, which avoids the complexity of gradient propagation in multi-module joint training and ensures that each module fully learns its corresponding feature representation.

[0023] (3) This invention proposes an adaptive upsampling strategy based on sample difficulty. It quantifies the identification difficulty of malignant samples by using two indicators: temperature gradient complexity and heat distribution uniformity. It also adopts differentiated data augmentation combinations and upsampling multiples according to the difficulty level, which effectively solves the problem of imbalance between the number of benign and malignant samples.

[0024] (4) In the fusion analysis module, the present invention constructs a structured scoring system consisting of vascular abnormality score, high temperature zone asymmetry score, nipple asymmetry score and QR curve index value. When the classification result is malignant, the system automatically outputs abnormal source prompt information, providing clinicians with traceable and verifiable quantitative evidence, which significantly improves the medical interpretability of the intelligent analysis system.

[0025] (5) The present invention introduces a symmetry constraint training mechanism based on left and right mirror consistency in each stage of phased training. By utilizing the anatomical prior of bilateral breast symmetry, it effectively reduces the risk of false unilateral abnormalities caused by noise or artifacts and improves the system's ability to identify minor asymmetry and early unilateral lesions. Attached Figure Description

[0026] Figure 1 This is a schematic diagram of the overall architecture of the system according to an embodiment of the present invention. Detailed Implementation

[0027] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the protection scope of the present invention.

[0028] like Figure 1 As shown, the present invention provides a multimodal feature fusion and intelligent analysis system based on thermal tomography images. The system includes four core components: a preprocessing module, a segmentation module, a thermal tomography module, and a fusion analysis module. The modules are connected by data flow to form a complete processing link.

[0029] In the system workflow, the preprocessing module first receives the thermal tomography image (i.e., the raw thermal tomography image) and inverts its temperature values ​​to generate a negative image. The raw thermal tomography image and the negative image together constitute the multimodal input. The segmentation module takes the negative image as input and outputs a vascular mask, a breast effective region mask, a nipple mask, and a lesion location mask. The thermal tomography module takes the raw thermal tomography image and the lesion location mask as input. Based on the lesion location mask, it determines the analysis location and, combined with the biological heat transfer equation, inverts the surface temperature at the lesion location to obtain the QR curve features reflecting the internal heat source characteristics of the tissue at the lesion location. The fusion analysis module receives the three types of masks output by the segmentation module and the QR curve features output by the thermal tomography module. Through the vascular analysis submodule, the high-temperature zone asymmetric analysis submodule, the nipple asymmetric analysis submodule, and the QR curve analysis submodule, it performs multi-dimensional comprehensive analysis and finally outputs the benign / malignant classification result. The first three sub-modules are trainable sub-networks that can learn the correlation between abnormal features and malignant lesions during training. The QR curve analysis sub-module processes the QR curve features to obtain the QR curve index value and applies gradient truncation so that it participates in the fusion decision in a fixed input form without interfering with the gradient updates of other sub-modules.

[0030] In this embodiment, data acquisition utilizes thermal tomography equipment to acquire thermal tomographic images of the breast region. The image resolution is 480×640 pixels, with a temperature range of 20 to 40 degrees Celsius. A pseudo-color encoding method is used, where blue represents low-temperature areas and red and yellow represent high-temperature areas. The system performs quality control on the acquired thermal tomographic images, including image sharpness checks, temperature calibration, and patient status control, ensuring that the images are blur-free and noise-free. Calibration is performed using a standard temperature source to ensure that the patient is in a stable state during image acquisition.

[0031] In this embodiment, 500 benign samples and 100 malignant samples were collected clinically and divided into training, validation, and test sets in a 3:1:1 ratio. The training and validation sets were used for model training and hyperparameter tuning, while the test set was used to evaluate model performance. Data annotation was performed by experienced clinicians using a double-blind annotation method to ensure annotation quality, and professional medical image annotation software was used to provide pixel-level accurate annotations. Annotation categories included four types: effective breast region, vascular location, benign lesion location, and malignant lesion location.

[0032] I. Adaptive Upsampling Module Because of the significant imbalance between the number of benign samples (500 cases) and malignant samples (100 cases), direct training would cause the model to be biased towards the majority class. Therefore, in one embodiment of this invention, an adaptive upsampling module based on sample difficulty is designed to perform differentiated upsampling processing on malignant samples. The core idea of ​​this module is to provide higher upsampling folds and richer enhancement methods for malignant samples with indistinct features and prone to misclassification, thereby enabling the model to obtain more learning opportunities from difficult samples during training.

[0033] Specifically, this module takes thermal tomographic images of malignant samples from the training set as input and evaluates the recognition difficulty of each sample from two dimensions. The first dimension is temperature gradient complexity: the system calculates the temperature difference between adjacent pixels in the thermal tomographic image, statistically analyzes the distribution characteristics of the gradient magnitude, and normalizes it to the range of 0 to 1. The smoother the gradient change, the smaller the temperature difference between the lesion and normal tissue, and the higher the recognition difficulty. The second dimension is thermal distribution uniformity: the system calculates the variance of the temperature distribution in the image and normalizes it to the range of 0 to 1. This indicator reflects the degree of dispersion of the temperature distribution.

[0034] The assessment results from the two dimensions are weighted and fused to obtain a comprehensive difficulty score, calculated using the following formula: ; Wherein, D is the overall difficulty score, ranging from 0 to 1; G is the temperature gradient complexity, ranging from 0 to 1, obtained by normalizing the temperature gradient amplitude distribution characteristics between adjacent pixels; H is the thermal distribution uniformity, ranging from 0 to 1, obtained by normalizing the temperature distribution variance. In the formula, 0.6 and 0.4 are the weighting coefficients for temperature gradient complexity and thermal distribution uniformity, respectively, and their sum is 1. The design of this formula is based on the following: temperature gradient complexity reflects the clarity of lesion boundaries and has a more direct impact on recognition difficulty, therefore it is given a higher weight of 0.6; thermal distribution uniformity reflects the overall temperature distribution pattern and is given a weight of 0.4 as an auxiliary criterion. The above weights were determined through grid search on the pre-experiment validation set.

[0035] Based on the comprehensive difficulty score, the system classifies malicious samples into three levels: Low-difficulty samples (D < 0.3) are upsampled 1-2 times, primarily using geometric transformations (including random rotation, horizontal flipping, and vertical flipping); Medium-difficulty samples (D > 0.3 and < 0.7) are upsampled 3-4 times, combining geometric and color transformations (including brightness and contrast adjustment); High-difficulty samples (D > 0.7) are upsampled 5-6 times, employing a combination of geometric transformations, color transformations, and noise addition (including Gaussian noise and salt-and-pepper noise). The thresholds of 0.3 and 0.7 are determined as follows: On the pre-experiment validation set, possible quartile threshold combinations are traversed with a step size of 0.1. The F1 score of the upsampled training set on the validation set is used as the evaluation metric, and the threshold combination that maximizes the F1 score is selected. Experiments show that after adaptive upsampling, the total number of malignant samples increased from 100 to about 450, the ratio of benign to malignant samples improved from 5:1 to about 1.1:1, and the classification accuracy on the test set was improved by about 3 to 5 percentage points compared with simple random oversampling.

[0036] II. Preprocessing Module The preprocessing module is responsible for converting thermal tomography images into negatives, so that the original image and the negative together constitute a multimodal input. The generation of the negative is achieved through an adaptive negative conversion algorithm, and the specific steps are as follows.

[0037] First, the system extracts the human body region based on temperature values, scales the human body region of the image to occupy 60% of the image area, and then pastes the scaled region back onto a mask of the original image size. Next, it converts the temperature values ​​into a negative image: the system sets an initial threshold of 300, performs low-temperature truncation on the temperature values, and sets the truncation value uniformly for regions with temperature values ​​less than the difference between the maximum temperature value and the threshold, thus retaining only the effective information of the high-temperature regions. After truncation, the processed temperature value is subtracted from the truncation value, multiplied by 255, and divided by the threshold, linearly mapping the temperature difference to a grayscale value range of 0 to 255. Finally, subtracting the grayscale value from 255 reverses the temperature distribution, resulting in a negative image. In the negative image, the original low-temperature regions are displayed as white, and the original high-temperature regions are displayed as black.

[0038] To ensure the quality and contrast of the negative image, this invention introduces an adaptive threshold adjustment mechanism. The system sets a target grayscale value range (in this embodiment, the preset lower limit is 100 and the preset upper limit is 180). When the average grayscale value of the negative image is lower than the preset lower limit, the threshold is reduced by 20 and the negative image is recalculated; when the average grayscale value of the negative image is higher than the preset upper limit, the threshold is increased by 30 and the negative image is recalculated. This iterative process continues until the average grayscale value of the negative image meets the preset target range, thereby providing a stable contrast input for subsequent segmentation and fusion analysis. The reason for the asymmetry between the reduction (20) and increase (30) in the threshold adjustment is that when the grayscale value is low, a small reduction in the threshold can effectively improve the overall brightness; while when the grayscale value is high, a larger increase is needed to significantly reduce the brightness. The above parameters were determined through experimental calibration.

[0039] III. Segmentation Module In this embodiment of the invention, the segmentation module is composed of a first segmentation module and a second segmentation module cascaded together to form a cascaded processing flow of full-image coarse segmentation, effective breast region cropping, and lesion fine segmentation.

[0040] The first segmentation module employs an encoder-decoder network architecture. In this embodiment, the MobileNetV4_UNET network architecture is used to simultaneously segment three target regions: the vascular region, the effective breast region, and the nipple region. The system uses MobileNetV4 as the network backbone, achieving efficient feature extraction through depthwise separable convolutions and inverse residual structures while maintaining the network's lightweight characteristics. Based on this, the invention adds symmetrical upsampling layers to the MobileNetV4 backbone network to realize the encoder-decoder architecture. The first segmentation module adopts a multi-head output design, taking a negative image as input and simultaneously outputting vascular masks, effective breast region masks, and nipple masks through three parallel output branches. This allows for multi-task segmentation with a single network, avoiding the computational redundancy caused by training independent networks for each segmentation target.

[0041] After the first segmentation module outputs the effective breast region mask, the system performs breast region cropping: the minimum bounding rectangle is calculated using the effective breast region mask, the negative image is cropped, and the cropped image is normalized and scaled to a fixed size (256×256 pixels in this embodiment) as input to the second segmentation module. This cropping process ensures that the second segmentation module can focus on feature learning of the breast region and avoid interference from background regions in lesion location segmentation.

[0042] The second segmentation module also employs an encoder-decoder network architecture. Taking a cropped breast region image as input, it performs fine segmentation of the lesion location and outputs a lesion location mask. This mask, visualized during the inference phase, provides clinicians with spatial localization information of the lesions. The second segmentation module achieves efficient feature extraction through depthwise separable convolutions and inverted residual structures, and ensures effective transmission of detailed information through skip connections.

[0043] IV. Thermal Chromatography Module The thermal tomography module takes raw thermal tomographic images and lesion location masks as input, and analyzes them using a bio-heat transfer physics model, providing thermophysical features and basis for the fusion analysis module. This module uses the Pennes bio-heat transfer equation to describe heat conduction within tissues, in the form: ; in For biological tissue density, Specific heat capacity of biological tissue Thermal conductivity of biological tissue Let be the temperature field function of the body tissue. For blood perfusion rate, Blood density, For the specific heat capacity of blood, For blood temperature, The heat production rate of biological tissue volume is also known as the metabolic heat production rate.

[0044] thermal chromatography module obtains QR The curve process is as follows: First, the Pennes biological heat transfer equation is applied: Blood flow items and metabolic rate Merged into a unified internal heat source item Meanwhile, assuming thermal conductivity coefficient If is a constant, then the Pennes biological heat conduction equation simplifies to: When the temperature field reaches a steady state, that is, when the temperature does not change with time, then Therefore, the steady-state differential equation for heat conduction is: .in It is the Laplace operator. It is the temperature field function of the body tissue. It refers to the intensity of the internal heat source, that is, the heat source within the body. It is the derivative of the temperature field function with respect to time.

[0045] The diseased area of ​​the body will generate excess heat compared to the normal area, and is considered an internal heat source in the heat conduction equation. If the diseased area itself is not very large, or its size is negligible relative to its distance from the body surface, then the heat source of the diseased area can be considered a point heat source, written as... ,in for function, Indicates in There is a heat source, There is no heat source in the body. Therefore, the steady-state heat conduction equation for a body with a pathogenic heat source can be written as: .

[0046] Assuming heat from a point heat source is conducted from within the body to the body surface, neglecting the body's own temperature control characteristics and assuming no temperature limitations on the body surface, heat transfer occurs within a three-dimensional infinite space, exhibiting spherical symmetry. A spherical coordinate system is established, with the origin O located at the point heat source. The position of any point within the body and on the body surface can be used... It means that, among them This represents the radial distance from the point to the point heat source. Indicates the polar angle. Let represent the azimuth angle. Then the steady-state heat conduction equation with an internal heat source is: Solve the equation: 1) When When, the equation becomes: The solution is: ,in and All are constants; without loss of generality, take... ; 2) When When, for the equation: Integrating the volume on both sides, we get: The integral volume With the origin O as the center of the sphere, and arbitrarily small positive numbers A sphere with radius [a]. On the other hand, by Gauss's law: ,have to: , bring in From this equation, we obtain the solution to the steady-state heat conduction equation for a heat source within a point: . Since the temperature of the internal heat source is highest at the point perpendicular to the body surface, the origin is taken as the point of contact with the body surface. The distance is The location of any point on the body surface is Then the distance from any point to the heat source inside the body Therefore, the expression for the temperature distribution on the body surface is derived as follows: .

[0047] In practical applications, the temperature distribution on the body surface is known. Depth of heat source in the body and intensity The information is an unknown quantity. The specific steps to obtain information about the depth and intensity of internal heat sources from the temperature distribution on the body surface are as follows: For uniformly distributed tissues, body surface temperature is... The points are centrally symmetrically distributed. Therefore, in practical applications, it is sufficient to simplify the two-dimensional distribution to a one-dimensional distribution. To handle it.

[0048] Perpendicular points on the body surface Centered on, r When the value range is increased from small to large, different results are obtained by fitting the temperature curves. r The value corresponding to Value, i.e., QR curve. Because r The larger the value, the wider the corresponding range of body surface temperature, representing the better the fitted result. The deeper the value, the greater the depth of the heat source. Thus, the QR curve represents the distribution of heat sources layer by layer from the body surface inwards, with r=0 as the center point.

[0049] The module first uses a lesion location mask to determine the spatial location of the QR curve analysis. It extracts body surface temperature data at different radii *r* from thermal tomography images, constructing a temperature distribution sequence with the analysis region radius *r* as the abscissa. Subsequently, based on this spatial location, the module establishes a point heat source model and uses analytical and numerical inversion algorithms to invert the body surface temperature data to obtain the internal heat source intensity and three-dimensional temperature field, thus generating a QR characteristic curve with the analysis region radius *r* as the abscissa and the heat source intensity *q* as the ordinate—the QR curve feature. Combined with clinical statistical data, the QR curve distributions of benign lesions, malignant lesions, and inflammation show significant discriminatory power. The QR curve feature serves as one of the inputs to the fusion analysis module, participating in fusion decisions during forward propagation. Since the calculation process of the QR curve is based on rules or numerical solutions and is inherently non-differentiable, this invention applies gradient truncation (i.e., stopping gradient operations) to this branch, preventing gradient backpropagation during backward propagation. This ensures the overall trainability of the fusion analysis module while incorporating prior physical knowledge.

[0050] V. Fusion Analysis Module The fusion analysis module is the core component of this system. It is responsible for integrating feature information from the segmentation module and the thermal chromatography module, performing multi-dimensional comprehensive analysis, and outputting benign and malignant classification results. This module receives the vascular mask, effective breast region mask, and nipple mask output by the first segmentation module, as well as the QR curve features output by the thermal chromatography module, and processes them through four sub-modules.

[0051] The vascular analysis submodule employs a lightweight convolutional neural network architecture. Its input consists of a vascular mask output from the first segmentation module and the original thermal tomographic image. Vascular features are extracted through convolutional and fully connected layers, ultimately outputting a vascular abnormality score. The score ranges from 0 to 1, where 0 indicates normal vessels and 1 indicates the presence of significantly thickened vessels or abnormally distributed clusters of vessels. In thermal tomographic images of abnormal vessels, locally thickened vessels and disordered vessel courses are typically observed. The vascular analysis submodule precisely performs a quantitative assessment of these morphological abnormalities.

[0052] The high-temperature zone asymmetry analysis submodule also employs a lightweight convolutional neural network architecture. Its inputs are thermal tomography images of both breasts and the effective breast region mask output by the first segmentation module. This submodule identifies potential pathological asymmetries by comparing the distribution characteristics of high-temperature zones in both breasts. In cases of abnormally high-temperature zones, one breast typically exhibits significant localized high-temperature accumulation, creating a significant temperature difference with the contralateral side. The high-temperature zone asymmetry analysis submodule quantifies this type of bilateral temperature distribution asymmetry. This submodule outputs a high-temperature zone asymmetry score, ranging from 0 to 1, where 0 indicates normal bilateral high-temperature zone symmetry, and 1 indicates significant high-temperature zone asymmetry.

[0053] The nipple asymmetry analysis submodule adopts a lightweight convolutional neural network architecture. The input is the nipple mask output by the first segmentation module and the thermal tomographic images of the bilateral nipple regions. It compares and analyzes the temperature and morphological characteristics of the bilateral nipple regions and outputs a nipple asymmetry score, with a value range from 0 to 1, where 0 indicates that the bilateral nipples are symmetrical and normal, and 1 indicates that there is obvious nipple asymmetry.

[0054] The QR curve analysis submodule receives QR curve features generated by the thermal tomography module based on the lesion location mask and analyzes and evaluates the QR curve at the lesion location. Specifically, this submodule assesses the degree of heat source anomaly at the lesion location based on the morphological characteristics of the QR curve (including heat source intensity value and curve distribution) and outputs the QR curve index value. During forward propagation, it serves as a fixed input to the fusion layer for comprehensive judgment. During backward propagation, the system applies gradient truncation (gradient stopping) to this branch, preventing gradient back propagation to the QR curve analysis and upstream thermal tomography modules. The outputs of the fusion layer, vascular analysis submodule, high-temperature zone asymmetric analysis submodule, and nipple asymmetric analysis submodule in the fusion analysis module are jointly trained with the outputs of other differentiable branches, thereby achieving end-to-end trainability of the fusion analysis module under the premise that the QR curve is not differentiable.

[0055] The fusion analysis module ultimately integrates the outputs of the four sub-modules and obtains the benign / malignant classification result through the fusion layer. During the inference phase, when the classification result is malignant, the system outputs abnormality source prompts to the clinician based on whether the scores of each sub-module exceed the corresponding threshold (set to 0.5 in this embodiment): if the vascular abnormality score exceeds the threshold, it indicates an abnormality in the blood vessels; if the high-temperature area asymmetry score exceeds the threshold, it indicates asymmetry in the high-temperature area; if the nipple asymmetry score exceeds the threshold, it indicates asymmetry in the nipple; if the QR curve index value is abnormal, it indicates a corresponding thermophysical abnormality. The threshold of 0.5 is selected as follows: on the validation set, candidate thresholds in the 0 to 1 interval are traversed with a step size of 0.05, and the value that maximizes the sum of the recall rate and the correct rejection rate is selected as the final threshold.

[0056] VI. Phased Training Strategy This invention employs a phased training strategy, dividing the entire system's training process into three stages, each focusing on training a specific module. This design differs from end-to-end one-time joint training, ensuring that each module fully learns its corresponding feature representation while controlling gradient propagation complexity.

[0057] Phase 1: Training the first segmentation module. The training input is a negative image, and the training labels are manually labeled blood vessel masks, effective breast region masks, and nipple masks. The loss function is: ; in Loss due to vascular segmentation, Loss due to effective segmentation of the breast tissue area. For nipple region segmentation loss, all three methods employ the cross-entropy loss function. After training, all network parameters of the first segmentation module are fixed to provide stable segmentation results for subsequent stages.

[0058] The second stage involves training the second segmentation module. The effective breast region mask output from the first segmentation module (trained in the first stage) is used to crop and normalize the negative image, resulting in a breast region image as input. The training labels are manually labeled lesion location masks (which must be spatially aligned with the cropped image, i.e., the same cropping and scaling operations are performed on the mask). The loss function is: ; in For lesion location segmentation loss, the cross-entropy loss function is used. After training, all network parameters of the second segmentation module are fixed. The training hyperparameters are the same as in the first stage.

[0059] Phase 3: Training the fusion analysis module. This phase trains the vascular analysis submodule, the high-temperature region asymmetric analysis submodule, the nipple asymmetric analysis submodule, and the fusion layer. The QR curve index values ​​of the QR curve analysis submodule, after gradient truncation, are used as fixed inputs. The loss function is: ; in For vascular analysis loss, For asymmetric analysis of loss in the high-temperature region, For the nipple asymmetry analysis loss, all three methods employ the binary classification cross-entropy loss function. , and In this embodiment, the balancing weight coefficients for each loss are all set to 1.0. These weights can be further optimized through grid search on the validation set. The training labels for the vascular analysis submodule are manually labeled vascular abnormality labels (binary classification); the training labels for the high-temperature region asymmetric analysis submodule are manually labeled high-temperature region asymmetric labels (binary classification); and the training labels for the nipple asymmetric analysis submodule are manually labeled nipple asymmetric labels (binary classification).

[0060] VII. Symmetry-Constrained Training Mechanism To further leverage the medical priori knowledge of the left-right symmetry of the breast in terms of anatomical structure and thermal distribution, another embodiment of the present invention introduces a symmetry constraint training mechanism in each stage of the phased training strategy. Without altering the phased training process and the parameter freezing strategy for each module, symmetry constraints are applied to the feature representation and output results of each module. This mechanism is applied only to samples labeled as benign during the training phase, and does not enforce consistency between left and right predictions during the inference phase. This ensures that true unilateral lesions are not erroneously symmetricized, while fully utilizing the left-right consistency pattern of benign samples.

[0061] In the first stage of training the first segmentation module, for samples labeled as benign, the system constructs a mirror image of the human body along the midline based on the original thermal tomography image. The first segmentation module takes the original image and the mirror image as input, and outputs two sets of corresponding segmentation masks (vascular segmentation mask, effective breast region segmentation mask, and nipple region segmentation mask). The prediction result of the mirror image, after inverse coordinate transformation, should spatially highly overlap with the prediction result of the original image. This invention adds a consistency loss based on mask overlap (using the complementary value of the Dice coefficient) to the conventional segmentation loss. By minimizing this loss, the first segmentation module learns segmentation results with consistent left and right structures on benign samples, thus explicitly utilizing the prior knowledge of left-right symmetry of the breast to improve the stability and robustness of the segmentation results. For samples labeled as malignant, this consistency constraint is not applied; only the conventional segmentation loss is used for training, avoiding incorrect flattening of the true left-right differences.

[0062] In the second stage of training the second segmentation module, a similar strategy is adopted: for benign samples, input pairs of original breast region images and left and right mirror-image breast region images are constructed. The segmentation results of the mirror images are inversely transformed and aligned with the original results. The overlap difference between the two is calculated as a consistency loss, which, together with the lesion location segmentation loss, optimizes the second segmentation module. By introducing mirror consistency constraints during the training stage of the second segmentation module, this invention enables the lesion segmentation network to maintain consistency in left and right prediction behavior on benign samples, thereby suppressing false unilateral lesions caused by noise or local artifacts and improving the reliability of lesion segmentation results. For malignant samples, no left-right consistency constraints are imposed, preserving the abnormal information of the true lesion side.

[0063] In the third stage of training the fusion analysis module, since the parameters of the first and second segmentation modules are frozen, the system only uses their fixed output results to construct symmetry constraints: based on the effective breast region mask, the position of the human body's midline is automatically estimated, and the original thermal tomography image is divided into left and right sections along the midline, extracting image blocks of the left and right breast regions and their corresponding blood vessel and nipple masks respectively. For benign samples, in addition to the classification loss and the supervision loss of each submodule, a consistency loss based on the difference between left and right scores is added to constrain the scores of the left and right sides to remain close. For malignant samples, no left-right consistency constraint is applied, preserving the abnormal information of the side with the true lesion.

[0064] In this embodiment of the invention, symmetry constraints and mirror consistency regularization are adaptively applied only during the training phase based on the benign or malignant annotation information of the samples, and are used to optimize the network parameters of the first segmentation module, the second segmentation module, and the fusion analysis module respectively; during the inference phase, the system does not make symmetry predictions for any new input cases.

[0065] During the inference phase, the system processes the input thermal tomography images according to the following process: Step 1: The preprocessing module receives the thermal tomography image and generates a negative image through an adaptive negative conversion algorithm, which together with the original image constitutes a multimodal input.

[0066] Step 2: The first segmentation module takes the negative image as input and outputs the blood vessel mask, the effective breast region mask, and the nipple mask.

[0067] Step 3: Calculate the minimum bounding rectangle using the effective breast region mask, crop the negative image and normalize it to a fixed size to obtain the breast region image.

[0068] Step 4: The second segmentation module takes the breast region image as input, segments the lesion location and outputs the lesion location mask, and obtains the lesion location information after visualization.

[0069] Step 5: The thermal tomography module performs QR curve analysis based on the original thermal tomography image, lesion location mask, and biological heat transfer equation to obtain QR curve characteristics.

[0070] Step 6: The fusion analysis module integrates the results from the vascular analysis submodule, the high-temperature zone asymmetric analysis submodule, the nipple asymmetric analysis submodule, and the QR curve analysis submodule, and outputs the benign / malignant classification results.

[0071] Step 7: The system outputs the benign / malignant classification results and lesion location information. When the classification result is malignant, based on whether the vascular abnormality score, high-temperature zone asymmetry score, nipple asymmetry score, and QR curve index value exceed the corresponding threshold, the system outputs abnormality source prompt information, providing clinicians with specific and traceable analytical reference information.

[0072] The above are merely preferred embodiments of the present invention and are not intended to limit the invention. For those skilled in the art, the present invention can have various modifications and variations. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention. The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described; only preferred embodiments of the present invention are illustrated. The descriptions are relatively specific and detailed, but should not be construed as limiting the scope of the present invention. As long as the combination of these technical features does not contradict each other, it should be considered within the scope of this specification.

[0073] It should be noted that those skilled in the art can make various modifications and improvements without departing from the inventive concept, and these all fall within the scope of protection of this invention. Therefore, the scope of protection of this invention should be determined by the appended claims.

Claims

1. A multimodal feature fusion and intelligent analysis system based on thermal tomography images, characterized in that, It includes a preprocessing module, a segmentation module, a thermal chromatography module, and a fusion analysis module; The preprocessing module inverts the temperature values ​​of the thermal tomography image to generate a negative image, and the thermal tomography image and the negative image constitute a multimodal input; The segmentation module takes the negative image as input and outputs a blood vessel mask, a breast effective region mask, a nipple mask, and a lesion location mask. The thermal tomography module takes the thermal tomography image and the lesion location mask as input, and obtains the QR curve features for the lesion location through the biological heat transfer equation. The fusion analysis module receives the vascular mask, the effective breast region mask, the nipple mask, and the QR curve features. After processing by the vascular analysis submodule, the high-temperature region asymmetric analysis submodule, the nipple asymmetric analysis submodule, and the QR curve analysis submodule with gradient truncation, it outputs the benign / malignant classification result. The segmentation module includes a first segmentation module and a second segmentation module; The first segmentation module takes the negative image as input, adopts an encoder-decoder network architecture, and simultaneously outputs the blood vessel mask, the effective breast region mask, and the nipple mask through a multi-head output design; The second segmentation module takes the cropped image of the breast region as input, segments the lesion location, and outputs the lesion location mask; In the fusion analysis module: the vascular analysis submodule adopts a lightweight convolutional neural network architecture, taking the vascular mask and thermal tomography image output by the first segmentation module as input, and outputting a vascular abnormality score with a value range of 0 to 1; the high-temperature region asymmetry analysis submodule adopts a lightweight convolutional neural network architecture, taking the thermal tomography image of both breasts and the effective breast region mask output by the first segmentation module as input, and outputting a high-temperature region asymmetry score with a value range of 0 to 1; the nipple asymmetry analysis submodule adopts a lightweight convolutional neural network architecture, taking the nipple mask and thermal tomography image of both nipple regions output by the first segmentation module as input, and outputting a nipple asymmetry score with a value range of 0 to 1; the QR curve analysis submodule analyzes the QR curve features and outputs a QR curve index value.

2. The system according to claim 1, characterized in that, The preprocessing module generates the negative image by inverting the temperature values ​​through the following steps: First, the temperature values ​​are truncated at low temperatures, and regions with temperature values ​​less than the difference between the maximum temperature value and the threshold are uniformly set as truncation values ​​to retain high-temperature region information; then, the truncated temperature difference values ​​are linearly mapped to a grayscale value range of 0 to 255; and finally, the negative image is obtained by subtracting the grayscale value from 255. The preprocessing module also includes an adaptive threshold adjustment mechanism, which decreases the threshold when the average gray value of the negative image is lower than a preset lower limit, and increases the threshold when the average gray value of the negative image is higher than a preset upper limit, and iteratively adjusts until the average gray value of the negative image meets the preset target range.

3. The system according to claim 1, characterized in that, The system also includes an adaptive upsampling module. This module takes thermal tomographic images of malignant samples in the training set as input, calculates the temperature gradient amplitude distribution characteristics between adjacent pixels of each malignant sample as the temperature gradient complexity, calculates the temperature distribution variance as the thermal distribution uniformity, and weights and fuses the temperature gradient complexity and the thermal distribution uniformity to obtain a comprehensive difficulty score. Based on the comprehensive difficulty score, the malignant samples are divided into three levels: low difficulty, medium difficulty, and high difficulty. Low difficulty samples are upsampled by geometric transformation, medium difficulty samples are upsampled by a combination of geometric transformation and color transformation, and high difficulty samples are upsampled by a combination of geometric transformation, color transformation, and noise addition. The higher the difficulty level, the greater the upsampling factor, thereby obtaining a class-balanced training dataset.

4. The system according to claim 3, characterized in that, The formula for calculating the overall difficulty score is as follows: ； Wherein, D is the overall difficulty score, ranging from 0 to 1; G is the temperature gradient complexity, ranging from 0 to 1, obtained by normalizing the temperature gradient amplitude distribution characteristics between adjacent pixels; H is the heat distribution uniformity, ranging from 0 to 1, obtained by normalizing the temperature distribution variance. and The weighting coefficients and ;when When a sample is identified as low-difficulty, it is upsampled by 1 to 2 times and subjected to geometric transformation; when When a sample is classified as medium difficulty, it is upsampled by 3 to 4 times and combined with geometric and color transformations; when If a sample is identified as a high-difficulty sample, a combination of 5 to 6 times upsampling, geometric transformation, color transformation, and noise addition is employed.

5. The system according to claim 1, characterized in that, The method for cropping the breast region includes: calculating the minimum bounding rectangle using the breast effective region mask output by the first segmentation module; cropping the negative image and then normalizing its size to obtain a fixed-size image containing only the breast effective region as input to the second segmentation module, thus forming a cascaded segmentation process of full-image coarse segmentation, breast effective region cropping, and lesion fine segmentation.

6. The system according to claim 1, characterized in that, The system employs a phased training strategy, comprising three phases executed sequentially: The first phase trains the first segmentation module, and the loss function is... for: ； in Indicates loss of blood vessel segmentation. This indicates the loss of effective segmentation of the breast tissue. This represents the nipple region segmentation loss. After training, the network parameters of the first segmentation module are fixed. The second phase trains the second segmentation module, with the loss function being... : ； in The segmentation loss at the lesion location is used as the basis for fixing the network parameters of the second segmentation module after training. The third stage of training involves the vascular analysis submodule, the high-temperature region asymmetric analysis submodule, the nipple asymmetric analysis submodule, and the fusion layer within the fusion analysis module. The QR curve index values ​​of the QR curve analysis submodule, after gradient truncation, are used as fixed inputs, and the loss function... for: ； in Indicates loss in vascular analysis. This indicates the loss from asymmetric analysis in the high-temperature region. The loss for nipple asymmetry analysis is represented by the binary cross-entropy loss function. , and These are the balancing weighting coefficients.

7. The system according to claim 6, characterized in that, The system also includes a symmetry constraint training mechanism, which applies mirror consistency constraints to samples labeled as benign in each stage of the phased training strategy: During the training phase of the first segmentation module, an input pair is constructed between the thermal tomography image and the left and right mirror images about the human body's central axis. The mask overlap consistency loss is calculated between the segmentation output and the original segmentation output after the inverse mirror transformation. During the training phase of the second segmentation module, an input pair of the original breast region image and the left and right mirrored breast region images is constructed, and a mirror consistency loss is applied to the lesion location segmentation result. During the training phase of the fusion analysis module, scores are extracted from the left and right breast regions respectively. A consistency constraint on the scores of the left and right sides is applied to samples labeled as benign, while no symmetry constraint is applied to samples labeled as malignant.

8. The system according to claim 1, characterized in that, Both the first segmentation module and the second segmentation module use a lightweight mobile network as the encoder backbone network, and achieve feature extraction through depthwise separable convolution and inverse residual structure. A symmetrical upsampling layer is constructed on the backbone network to form an encoder-decoder network architecture. During the inference phase, the system outputs the benign / malignant classification result by the fusion analysis module, and the second segmentation module outputs the lesion location information through result visualization. When the benign / malignant classification result is abnormal, the system outputs an abnormality source prompt information based on whether the vascular abnormality score, the high temperature zone asymmetry score, the nipple asymmetry score, and the QR curve index value exceed the corresponding threshold.

Images