Multi-modal image fusion nasolacrimal duct obstructive disease diagnosis and grading system

Abstract

The invention relates to the technical field of medical image processing, and discloses a multi-modal image fusion nasolacrimal duct obstructive disease diagnosis and grading system which comprises the following steps: acquiring head CT data, performing space standardization, and generating osseous channel and soft tissue channel input tensors based on double-window parameters; a double-flow attention interaction network is constructed, a spatial attention mask is generated by using a bony feature extraction branch, and weighted fusion is performed on the pathological features of the soft tissue through a topology guide attention unit; a semi-supervised model training module is adopted, a consistency constraint strategy is executed on unlabeled samples, and network parameters are updated by minimizing a composite loss function; and finally outputting disease property classification and blocking degree grading results. According to the method, the bony anatomical structure and the soft tissue pathological information can be effectively combined, the model performance is improved by using the unlabeled data, and high-precision automatic diagnosis and grading of nasolacrimal duct blockage are realized.

Description

Technical Field

[0001] This invention relates to the field of medical image processing technology, specifically to a multimodal image fusion-based diagnostic and grading system for nasolacrimal duct obstructive diseases. Background Technology

[0002] Nasolacrimal duct obstruction is a common lacrimal duct disease in ophthalmology, leading to symptoms such as epiphora and acute and chronic dacryocystitis, which seriously affect patients' quality of life. In clinical diagnosis and treatment, computed tomography (CT) scans, especially lacrimal duct CT angiography, are important imaging evidence for determining the location, nature, and degree of obstruction. However, the anatomical structure of the nasolacrimal duct region is complex, and there are significant differences in density between the bony passage and the internal soft tissues (such as mucosa, obstruction material, and contrast agent).

[0003] Current technologies for processing such image data using deep learning models typically use images with a single window width and level as input, or simply normalize CT values. This approach struggles to balance the localization information of bony structures with the textural details of soft tissue lesions. If the focus is on displaying bony structures through bone windows, the contrast of soft tissues within the lumen becomes too low, making it difficult to distinguish between mucosal thickening and fluid filling. Conversely, if the focus is on displaying soft tissues through soft tissue windows, the surrounding high-density bone can create artifacts or obscure anatomical boundaries, causing the model to lose crucial diagnostic information during feature extraction.

[0004] Furthermore, the nasolacrimal duct is a narrow space adjacent to complex tissues such as the maxillary sinus, nasal turbinate, and orbit. Most current automated diagnostic algorithms employ general convolutional neural network architectures, lacking attention mechanisms specific to the duct's topology. When extracting features, the network often processes all regions equally, making it susceptible to interference from surrounding irrelevant anatomical structures or noise, leading to inaccurate assessments of the degree of stenosis. Simultaneously, accurate annotation of medical images requires significant expert effort, and existing methods largely rely on fully supervised learning. When labeled data is scarce, the model's generalization ability and robustness often fail to meet the demands of practical clinical applications. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides a multimodal image fusion-based diagnostic and grading system for nasolacrimal duct obstructive diseases. This system solves the problems of low diagnostic accuracy and imprecise grading in existing technologies when processing nasolacrimal duct CT images. These problems stem from the difficulty in simultaneously locating bony structures and extracting soft tissue pathological features, as well as the susceptibility to interference from the surrounding complex anatomical environment and the high cost of annotation.

[0006] To achieve the above objectives, the present invention provides the following technical solution: This invention provides a multimodal image fusion-based diagnostic and grading system for nasolacrimal duct obstructive diseases.

[0007] The system includes a data acquisition and multi-window preprocessing module, a dual-stream attention interaction network construction module, a semi-supervised model training module, and an intelligent diagnosis and hierarchical output module.

[0008] In the data acquisition and multi-window preprocessing stage, the system acquires computed tomography (CT) scan data of the subject's head. To eliminate differences in slice thickness and resolution caused by different acquisition devices, the data is first unified to a standard anatomical space using a rigid registration algorithm. The line connecting the anterior and posterior commissures is selected as a reference plane for rotational correction, and a voxel block containing the bilateral lacrimal duct system is cropped with the midpoint of the line connecting the bilateral medials as the center, and resampled to a uniform spatial resolution. Based on this, the system generates bony channel input tensors and soft tissue channel input tensors according to preset dual-window parameters.

[0009] The specific tensor generation logic is as follows: For the bony channel input tensor, the voxel value is subtracted from the lower limit of the bone window, then divided by the bone window width, and the result is truncated and mapped to the 0-1 interval to suppress soft tissue background and highlight bony anatomical structures; for the soft tissue channel input tensor, the voxel value is subtracted from the lower limit of the soft tissue window, then divided by the soft tissue window width, and the result is truncated and mapped to the 0-1 interval to stretch the grayscale contrast of the lacrimal sac mucosa and the filling material within the lumen. The lower limit of each window is calculated as the window level minus half the window width.

[0010] In the network construction phase, the system constructs a dual-stream attention interaction network, which includes a bony feature extraction branch and a soft tissue feature extraction branch, used to extract bony topological features and soft tissue pathological features, respectively. The system utilizes topology to guide the attention unit to perform cross-modal weighted interactions. Specifically, this unit uses a 3D convolutional layer to compress the multi-channel bony topological features output by the bony feature extraction branch into a single-channel feature map; it then uses a sigmoid activation function to map this single-channel feature map into a spatial attention mask with values ​​between 0 and 1, representing the probability response intensity of each spatial location belonging to a bony lacrimal duct structure; subsequently, it performs element-wise multiplication of this spatial attention mask with the soft tissue pathological features, and performs element-wise addition of the result with the original soft tissue pathological features via residual connection, thereby generating a fused feature map. This mechanism uses relatively fixed bony structures to guide the model to focus on soft tissue lesions within the lumen.

[0011] In the diagnostic output stage, the system performs global average pooling on the fused feature map through a fully connected classification layer, compressing the three-dimensional feature tensor into a one-dimensional feature vector. This vector is then input into a multilayer perceptron network containing two sets of independent output nodes, outputting the classification probabilities of primary and secondary obstruction, and the grade probabilities of patency, mild stenosis, and severe obstruction. The system selects the category with the highest probability value as the final structured diagnostic result. Furthermore, the system is equipped with a visualization heatmap generation unit. It upsamples the spatial attention mask using bilinear interpolation and overlays it onto the grayscale image of the soft tissue channel using alpha blending technology, visually displaying the model's area of ​​interest.

[0012] This invention provides a multimodal image fusion-based diagnostic and grading system for nasolacrimal duct obstructive diseases. It offers the following advantages: 1. This invention, through data acquisition and multi-window preprocessing modules, generates independent input channels for bony anatomical structures and soft tissue pathological features, solving the problem that traditional single-window CT images cannot simultaneously and clearly display bony canals and their internal soft tissue lesions. This dual-channel parallel input mechanism can preserve the subject's original anatomical information to the greatest extent, enabling the model to both locate anatomical structures using bony channels and identify mucosal thickening or obstructions using soft tissue channels, significantly enriching the dimensions of diagnostic features.

[0013] 2. This invention constructs a dual-stream attention interaction network, utilizing topology-guided attention units to achieve deep interaction of cross-modal features. This mechanism uses the relatively fixed topological structure of bony channels to generate a spatial attention mask, weighting and focusing features of soft tissue channels. This design guides the network to automatically suppress background noise interference from surrounding irrelevant tissues such as the maxillary sinus and eyeball, forcing the model to focus on subtle pathological changes within the nasolacrimal duct, thereby effectively improving the accuracy of locating and identifying stenotic and obstructive lesions.

[0014] 3. This invention introduces a semi-supervised model training module and a consistency constraint strategy, effectively alleviating the problem of scarcity of high-quality labeled data in the field of medical imaging. By imposing consistency constraints on a large number of unlabeled samples and utilizing the data distribution characteristics of unlabeled data to correct the model's decision boundary, this not only reduces the model's dependence on large-scale expert-labeled data but also enhances the model's robustness and generalization ability when facing scanning data from different devices, making it more adaptable to actual clinical application scenarios. Attached Figure Description

[0015] Figure 1 This is a schematic diagram of the system architecture of the present invention; Figure 2 This is a flowchart illustrating the data acquisition and multi-window preprocessing module of the present invention. Figure 3This is a flowchart of the expert collaborative verification and incremental training mechanism of the present invention. Detailed Implementation

[0016] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0017] Please see the appendix Figure 1-3 This invention provides a multimodal image fusion-based diagnostic and grading system for nasolacrimal duct obstructive diseases, comprising: a data acquisition and multi-window preprocessing module, a dual-stream attention interaction network construction module, a semi-supervised model training module, and an intelligent diagnostic and grading output module.

[0018] This invention provides a method for diagnosing and grading nasolacrimal duct obstructive diseases using multimodal image fusion, comprising the following steps: Three-dimensional computed tomography data of the nasolacrimal duct region were acquired, and the scan data were decoupled into two channels to generate bony channel input tensors and soft tissue channel input tensors. Construct a two-stream attention interaction network that includes a bony feature extraction branch, a soft tissue feature extraction branch, and a topology-guided attention unit; The bony channel input tensor and the soft tissue channel input tensor are respectively input into a two-stream attention interaction network. The spatial attention mask generated by the bony channel input tensor is used to perform weighted interaction on the features extracted by the soft tissue channel input tensor to generate a fused feature vector. A hybrid dataset containing labeled and unlabeled samples is constructed, and the parameters of the two-stream attention interaction network are optimized using a semi-supervised training strategy. The semi-supervised training strategy includes calculating the supervision loss of labeled samples and the consistency loss of unlabeled samples. The data of the nasolacrimal duct region to be diagnosed is input into the trained and optimized dual-stream attention interaction network, which outputs the qualitative category and grading results of nasolacrimal duct obstructive disease.

[0019] In the specific implementation of the data acquisition and multi-window preprocessing module, the acquisition and standardization of raw image data are fundamental steps to ensure the accuracy of subsequent models.

[0020] Computed tomographic (CT) scans of the subject's head were acquired. The data was obtained from a medical CT scanner, covering the area from the upper edge of the eyebrow to the lower opening of the nasolacrimal duct. To ensure clear visualization of the nasolacrimal duct's fine structures, the scanning parameters were set to high resolution mode, with slice thickness and interslice spacing both set to 1.25 mm or less, tube voltage set to 120 kV, and tube current adjusted automatically using milliampere-second (mA-second) technology. After scanning, the original projection data was reconstructed into 3D volumetric data in DICOM format using a standard algorithm. This 3D volumetric data contains complete anatomical information of the bilateral orbits, nasal cavities, and lacrimal systems.

[0021] The acquired 3D volumetric data underwent spatial standardization. Since head positioning may vary among patients during scanning, a rigid registration algorithm was used to unify all 3D volumetric data to a standard anatomical space. The line connecting the anterior and posterior commissures was selected as a reference plane, and rotation and translation corrections were applied to the images to ensure the anatomical axis of the nasolacrimal duct remained consistent in 3D space. Subsequently, according to preset region of interest (ROI) cropping rules, 3D voxel blocks containing the complete bilateral lacrimal duct system were cropped, centered on the midpoint of the line connecting the bilateral medials. The cropped voxel blocks were resampled to a uniform spatial resolution, for example, each voxel corresponding to an actual physical size of 0.5mm × 0.5mm × 0.5mm, to eliminate the influence of differences in acquisition resolution from different devices.

[0022] Intensity normalization was performed on the spatially standardized data. Voxel values ​​in CT images are expressed in Heinz units (HU). While HU values ​​from different devices are physically comparable, systematic biases still exist. A truncation normalization strategy was adopted, setting the effective HU value range to [-1000, 3000]. Voxel values ​​below -1000 HU were truncated to -1000, and voxel values ​​above 3000 HU were truncated to 3000. The truncated values ​​were then linearly transformed to the [0,1] interval to generate standardized 3D input data. This step effectively suppresses the interference of metal artifacts and in vitro background noise on subsequent feature extraction, ensuring the consistency of the data distribution input to the network.

[0023] After completing the spatial standardization and normalization of the data, the system performs channel decoupling processing based on dual-threshold mapping. This processing aims to separate single-modality CT scan data into two independent feature input channels, namely the bony channel and the soft tissue channel, according to the physical density characteristics of different tissues, so that the subsequent network can learn the anatomical topology and lesion tissue features respectively.

[0024] A bony channel input tensor is generated. The system retrieves preset bone window parameters, including the bone window level and width. The bone window level is typically set between 300 HU and 500 HU, and the bone window width is set between 1500 HU and 2000 HU. A linear mapping function is used to transform the voxel values ​​of the original CT data. Specifically, for each voxel, its gray value is subtracted from half the difference between the bone window level and the bone window width, and then divided by the bone window width. If the calculation result is less than 0, it is assigned a value of 0; if the calculation result is greater than 1, it is assigned a value of 1; if the calculation result is between 0 and 1, the calculated value is retained. After the above processing, soft tissue, fat, and air components below the lower limit of bone density in the original data are suppressed to low background pixel values, while high-density bony nasolacrimal duct walls, nasal bones, and surrounding bone structures are mapped to high-contrast foreground features, thus forming a bony channel input tensor that retains only anatomical topological information.

[0025] A soft tissue channel input tensor is generated. The system retrieves preset soft tissue window parameters, including the soft tissue window level and width. The soft tissue window level is typically set between 40 HU and 60 HU, and the soft tissue window width is set between 300 HU and 400 HU. Using the same linear mapping logic, the voxel values ​​of the original CT data are mapped to the [0,1] interval. Under these parameter settings, high-density bone areas are displayed as saturated bright pixels, while air is displayed as dark pixels. The lacrimal sac mucosa, luminal effusion, inflammatory proliferative tissue, and possible obstructions, which are mainly in the intermediate density range, are stretched to a high-contrast grayscale range. This process highlights the texture and morphological details of the diseased tissue, forming the soft tissue channel input tensor.

[0026] A dual-channel parallel input data structure is constructed. The generated bony channel input tensor and the generated soft tissue channel input tensor are combined. Although both originate from the same set of CT scan data, their physical properties do not interfere with each other. The bony channel input tensor serves as the anatomical modality providing spatial geometric constraints in the subsequent network, while the soft tissue channel input tensor serves as the pathological modality providing the basis for pathological diagnosis. This decoupling process transforms the traditional single-channel grayscale image into a dual-channel tensor containing clear anatomical and pathological semantics, providing a physically separated data foundation for feature extraction in the subsequent two-stream network. The specific algorithmic implementation of the above linear mapping and window width / level adjustment are standard techniques in the field of digital image processing, and their mathematical derivation will not be elaborated here.

[0027] In the dual-stream attention interaction network construction module, in order to simultaneously capture the rigid anatomical structure of the nasolacrimal duct and the subtle lesions of its internal soft tissue, the system constructs a feature extraction architecture with two parallel processing paths: a bony feature extraction branch and a soft tissue feature extraction branch. Although these two branches maintain spatial alignment in physical structure, they are independent in parameter learning to adapt to the differences in feature distribution of different modalities.

[0028] A bony feature extraction branch is constructed. This branch is configured to receive the bony channel input tensor output by the aforementioned preprocessing module. The backbone network of this branch adopts a three-dimensional convolutional neural network structure, or a Transformer-based three-dimensional visual encoder structure. In specific implementations, this backbone network consists of several cascaded convolutional modules, each containing a three-dimensional convolutional layer, a batch normalization layer, and a nonlinear activation layer. The kernel size of the three-dimensional convolutional layer is typically set to 3×3×3, and the stride is set to 1 or 2, to extract local geometric features while preserving spatial depth information. This branch transforms the input bony image into a high-dimensional feature representation, i.e., a bony feature map, through layer-by-layer abstraction. This bony feature map mainly encodes the topological information of the nasolacrimal duct, such as its bony boundaries, duct orientation, diameter variations, and bone continuity, serving as a guide source for subsequent attention mechanisms.

[0029] A soft tissue feature extraction branch is constructed. This branch is configured to receive the soft tissue channel input tensor. To ensure a one-to-one correspondence of spatial locations during subsequent feature interactions, the soft tissue feature extraction branch uses the exact same network layer depth and downsampling ratio as the bony feature extraction branch. For example, if the bony branch performs three downsampling operations to achieve a feature map spatial resolution of one-eighth of the original input, the soft tissue branch will also perform three downsampling operations. Despite structural isomorphism, the soft tissue feature extraction branch has its own independent set of weight parameters, which are not shared with the bony branch. This branch focuses on learning the grayscale changes under the soft tissue window, extracting pathological features characterizing the size of the lacrimal sac, the density of fluid accumulation within the lumen, the degree of mucosal thickening, and the texture of obstructions, and generating a soft tissue feature map.

[0030] Spatial alignment and dimensionality unification of the feature maps are performed. At the ends of the two branches, the bone feature maps and soft tissue feature maps output by the network have the same spatial dimensions (depth, height, width) and the same number of channels. Regarding the specific selection of the backbone network, those skilled in the art can choose existing 3D feature extraction network architectures such as ResNet-3D, DenseNet-3D, or Semiformer as the basic backbone based on computational resources and accuracy requirements. Any network structure capable of mapping from 3D volume data to a high-dimensional feature space falls within the scope of this invention. This dual-stream parallel design fundamentally avoids the drawback of single-stream networks, which tend to ignore subtle soft tissue lesions and overemphasize high-contrast bone structures when processing mixed modalities.

[0031] Building upon dual-stream feature extraction, this invention introduces a topology-guided attention unit to address the challenge of accurately locating soft tissue lesions in complex anatomical contexts. This unit utilizes the anatomical location information inherent in bony features to generate a spatial constraint mechanism, enabling the network to automatically focus on soft tissue signals within bony channels while ignoring interference from external, irrelevant tissues.

[0032] Generate a spatial attention mask. First, obtain the bone feature map output by the bone feature extraction branch. Since this feature map has multiple feature channels, and each channel may respond to different skeletal texture features, it needs to be compressed into a single-channel spatial weight map. A convolutional kernel with a size of [missing information] is used. A three-dimensional convolutional layer is used to perform channel dimensionality reduction on the bony feature map. This convolutional operation aggregates multi-channel information into a scalar value, which reflects the probability response intensity of each spatial location belonging to a bony lacrimal duct structure. Subsequently, a sigmoid activation function is used to nonlinearly map the value of the convolution output to the range (0,1), generating a spatial attention mask. The mask is spatially the same dimension as the feature map, and its numerical value intuitively represents the importance weight of the corresponding spatial location: the weight approaches 1 in the bony channel and edge region; and the weight approaches 0 in the background region far away from the bony channel.

[0033] Perform weighted interaction of performance features. Obtain the soft tissue feature map output by the soft tissue feature extraction branch. Utilizing the generated spatial attention mask Element-wise multiplication is performed on the soft tissue feature map. Specifically, for each voxel feature vector in the soft tissue feature map, it is multiplied by the mask weight value of the corresponding spatial location. This operation is essentially a spatial filtering of the soft tissue features based on anatomical topology: lesion features located on the nasolacrimal duct anatomical path (such as luminal obstruction or stenosis) are preserved or enhanced, while artifacts or irrelevant soft tissue features located outside the path are suppressed or filtered out.

[0034] Residual features are fused. To prevent the attention mechanism from over-suppressing potentially weak lesion signals or causing the loss of original feature information, a residual connection structure is introduced on top of the weighted interaction. The processed weighted feature map is then combined with the original soft tissue feature map. Element-by-element addition is performed to obtain the final fused feature map. This fused feature map not only includes all the rich soft tissue details in the original input, but also highlights the pathological information of key anatomical regions through bony topology guidance, achieving deep synergy between structural and pathological modalities at the feature level. This design simulates the cognitive process of clinicians when reviewing images: first locating the lacrimal duct through the bone window, and then observing the internal condition of that location under the soft window.

[0035] After the interaction and enhancement of multimodal features are completed through topology-guided attention units, the system needs to transform the high-dimensional three-dimensional feature tensor into specific medical diagnostic conclusions. This section elaborates on the dimensionality reduction processing and classification prediction mechanism after feature fusion.

[0036] Perform global feature aggregation. The fused feature map is output from the previous layer network. The feature map is a four-dimensional tensor (number of channels × depth × height × width). While it contains rich spatial details, its high dimensionality makes it unsuitable for direct classification decisions. The system uses a global average pooling (GAP) layer to process this feature map. The GAP operation averages all spatial pixel values ​​within each feature channel, compressing the information of the entire three-dimensional space into a single value. After this operation, the originally massive four-dimensional feature tensor is transformed into a one-dimensional feature vector. The length of this feature vector is equal to the number of channels in the feature map, and it highly condenses the comprehensive pathological representation of the entire nasolacrimal duct region under anatomical constraints.

[0037] A classification decision network is constructed. The generated feature vectors are input into a fully connected layer network. This fully connected layer network acts as a classifier and can be composed of a single-layer or multi-layer perceptron. In a multi-layer structure, ReLU activation functions and Dropout layers can be added between layers to increase non-linear fitting ability and prevent overfitting. The number of output nodes in the fully connected layer is set according to the specific diagnostic task. In this embodiment, the output nodes are configured in two groups: the first group is used to output the classification of the disease nature (e.g., primary obstruction, secondary obstruction), and the second group is used to output the grading of the obstruction degree (e.g., mild stenosis, severe obstruction).

[0038] Generate a probability distribution output. The output values ​​of the classification decision network are processed by the Softmax normalization function to transform them into a probability distribution. For each diagnostic category, the Softmax function calculates its corresponding confidence score, with the sum of all scores equal to 1. The system selects the category with the highest probability value as the final prediction result. For example, if the probability value of the "severe obstruction" category in the "degree grading" output group is 0.85, exceeding the probability values ​​of other categories, the system determines the case as severe obstruction. The initialization of weight parameters and the backpropagation update algorithm in the classification network are common techniques in the field of deep learning and will not be elaborated here. Through the above settings, this system can automatically map complex anatomical and pathological image features into structured text results that conform to clinical diagnostic criteria.

[0039] To overcome the problem of scarcity of high-quality labeled data in the field of medical imaging, this invention adopts a semi-supervised learning strategy in the model training stage. This requires first constructing a hybrid dataset that contains both a small number of accurately labeled samples and a large number of unlabeled samples, and designing a corresponding sampling mechanism to ensure the stability of training.

[0040] A labeled dataset was constructed. Historical images of nasolacrimal duct obstruction diseases, jointly diagnosed by senior radiologists and ophthalmologists, were collected. Each image was accompanied by a clear diagnostic label, including the nature of the disease (primary or secondary) and the degree of obstruction (stenosis or blockage). The aforementioned data preprocessing procedures were performed on these image data to generate standardized two-channel input tensors. The processed data and its corresponding label set were defined as the labeled dataset. Although this dataset is relatively small, it provides a definite mapping relationship between pathological features and diagnostic conclusions, serving as a fundamental source of supervision for model learning.

[0041] Construct an unlabeled dataset. Export CT image data conforming to the nasolacrimal duct scanning protocol but not yet finely annotated manually from the hospital's image archiving system in batches. This data also underwent a standardized preprocessing workflow, but did not include diagnostic labels. This dataset is defined as the unlabeled dataset. Unlabeled datasets are typically much larger than labeled datasets, containing extensive information on the distribution of anatomical variations and pathological manifestations in the population, which helps models learn generalized representations of features.

[0042] Perform balanced mixture sampling. In each iteration of model training, it is necessary to perform balanced mixture sampling from... and Samples are drawn from each dataset to form a training batch. To avoid the model becoming overly biased towards fitting a small amount of labeled data or being dominated by a large amount of unlabeled data, the system employs a fixed sampling ratio strategy. A hyperparameter is set. This represents the ratio of unlabeled samples to labeled samples. For a size of... For labeled sample batches, the system accordingly extracts samples of size [size missing]. Unlabeled sample batches. Typically, The value ranges from 1 to 7. This sampling method ensures that the model receives both strong supervision signals from the gold standard label and distribution consistency signals from unlabeled data during each parameter update, maintaining a dynamic balance between supervised and unsupervised learning. The specific data loading and batch generation implementation can be accomplished using the data loader interface of existing deep learning frameworks; code details will not be elaborated here.

[0043] To fully leverage the value of unlabeled data, this invention, based on the FixMatch semi-supervised framework, designs a strong-weak enhancement strategy based on anatomical consistency, specifically tailored to the characteristics of nasolacrimal duct CT images. The core of this strategy is that for the same unlabeled image, regardless of the degree of interference applied, as long as its anatomical structure remains intact, the model's diagnostic results should remain consistent.

[0044] Perform weak augmentation operations. For each unlabeled sample... The system first applies a weak enhancement function to it. This function only includes geometric transformations that do not alter the semantic content of the image. Specific operations include: horizontal flipping with a 50% probability to simulate left-right lateral differences; and random translation in three-dimensional space, with the translation amplitude controlled within 10% of the image size, to simulate slight postural deviations during patient scanning. The sample, after weak enhancement, is input into the model, which predicts its class probability distribution under the current parameter state. The class with the highest probability is selected as the pseudo-label for the unlabeled sample, provided that this probability value exceeds a preset confidence threshold to ensure the reliability of the pseudo-label.

[0045] Perform strong enhancement operations. For the same unlabeled sample The system applies strong enhancement functions in parallel. Unlike general natural image enhancement, the strong enhancement here aims to simulate artifacts and pathological variations commonly found in CT imaging. The specific operation includes two parts: first, random noise injection, adding additive noise conforming to a Gaussian distribution to the image to simulate quantum noise during low-dose scanning; second, anatomical structure occlusion, randomly selecting rectangular regions in the image and setting their pixel values ​​to zero, forcing the model to not rely on a single local feature for judgment. Furthermore, random gamma transforms can be introduced to non-linearly adjust image contrast, simulating the differences in reconstruction algorithms across different CT equipment.

[0046] Consistency constraints are established. The strongly enhanced samples are input into the model to obtain its predictions. The system does not directly compare these predictions with the true labels (because there are no labels), but instead forces the predictions to remain consistent with the generated pseudo-labels. This mechanism is called consistency regularization. Its technical principle is that if the model truly learns the essential pathological features of nasolacrimal duct obstruction (such as bony stenosis or soft tissue filling), then the model's judgment should remain stable regardless of whether there is noise interference or local occlusion in the image. Through this "strong interference input - weak interference supervision" training mode, the model is forced to search for robust anatomical and pathological features in complex noisy environments, thereby significantly improving generalization performance by utilizing a large amount of unlabeled data.

[0047] In the semi-supervised training process, in order to synergistically utilize the precise supervision information of labeled data and the consistency constraint information of unlabeled data, this invention constructs a composite total loss function that includes a supervised loss component and an unsupervised consistency loss component. The network parameters are iteratively updated by minimizing this total loss function through the gradient descent algorithm.

[0048] The supervised loss for labeled samples is calculated. For each labeled sample in a training batch, the model outputs a predicted probability distribution. The system obtains the corresponding true diagnostic label for that sample. The cross-entropy loss function is used to measure the difference between the model's predicted distribution and the true label distribution. Specifically, the cross-entropy loss is achieved by calculating the negative of the log probability of the true class. The closer the model's predicted probability of the true class is to 1, the closer the loss value is to 0; conversely, the lower the predicted probability, the larger the loss value. This supervised loss component directly constrains the model to learn the correct mapping relationship between pathological features and diagnostic categories, ensuring that the convergence direction of the model training conforms to medical diagnostic criteria.

[0049] Calculate the consistency loss for unlabeled samples. For unlabeled samples in the same batch, as before, the system generates pseudo-labels through weak augmentation prediction and obtains the probability distribution to be verified through strong augmentation prediction. The system only selects samples whose weak augmentation prediction confidence exceeds a preset threshold (e.g., 0.95) to participate in the loss calculation. For these selected samples, the difference between their strong augmentation prediction distribution and the pseudo-label is calculated using the cross-entropy loss function. This loss component is called the unsupervised consistency loss. It does not rely on manual annotation but on the predictive stability of the model itself, forcing the model to maintain output consistency when faced with image noise and perturbations, thereby mining the intrinsic structural features of the data.

[0050] The overall optimization objective is calculated. The supervised loss and unsupervised consistency loss are weighted and summed to obtain the final total loss function used for backpropagation. During the summation process, a dynamic weight coefficient is introduced to adjust the proportion of unsupervised loss. In the early stages of training, because the model is not yet stable and the quality of pseudo-labels is low, this weight coefficient is set relatively small, and training is dominated by labeled data. As the training epochs increase, model accuracy improves, pseudo-labels become more reliable, and the weight coefficient gradually increases, allowing unsupervised data to play a greater corrective role. Through this dynamic weighting mechanism, the system can maximize the use of mixed datasets for knowledge distillation while ensuring training stability. The derivative of the loss function and the parameter update process based on stochastic gradient descent or the Adam optimizer are standard mathematical principles in the field of deep learning and will not be elaborated upon here.

[0051] After the deep neural network model completes semi-supervised training and reaches convergence, the system enters the inference application stage. The core task of this stage is to apply the trained and solidified model parameters to new, unseen patient image data to generate objective diagnostic conclusions.

[0052] The system loads model parameters and diagnostic data input. It initializes the dual-stream attention interaction network architecture in the computing device's memory and reads the weight file generated during training to restore the model parameter state. It acquires the raw nasolacrimal duct CT data of the patient to be diagnosed, executes the standardized process defined in the data acquisition and multi-window preprocessing module, and generates bony channel input tensors and soft tissue channel input tensors. To ensure consistency in inference, the resampling resolution parameters, normalized truncation threshold parameters, and window width / level mapping functions used in the inference phase are exactly the same as those used in the training phase.

[0053] Forward propagation inference is performed. The preprocessed bony channel input tensor and soft tissue channel input tensor are fed into the deep network in parallel. The network performs convolution, pooling, and activation operations on the input tensors according to a predefined hierarchical structure. Inside the network, topology-guided attention units calculate and generate a spatial attention mask based on the bony feature map and perform element-wise weighted operations on the soft tissue feature map. The entire process is in inference mode, performing only forward computation and without gradient backpropagation or parameter updates. The network outputs a two-dimensional probability distribution vector, corresponding to disease nature classification and obstruction degree grading, respectively.

[0054] Results Decoding and Confidence Determination. The system numerically decodes the output probability distribution vector. For each classification dimension, the category corresponding to the index with the highest probability value is selected as the prediction result, and this highest probability value is used as the diagnostic confidence score. The system has an internal safety threshold (e.g., 0.85). The comparator compares the highest confidence score of the model output with this threshold. If it is lower than this threshold, the system generates an "uncertain" flag and activates a manual review trigger signal.

[0055] Generate a visual attention heatmap. The system extracts intermediate layer features from the output of the topology-guided attention unit, i.e., the spatial attention mask. A bilinear interpolation algorithm is used to upsample the spatial dimensions of this mask to match the original CT image. The upsampled mask values ​​are mapped onto a preset pseudo-color spectrum to generate a color heatmap. The system uses alpha blending technology to synthesize the color heatmap with the original soft tissue window grayscale image, setting the blending weight coefficient to 0.3 to 0.5. This ensures that the generated image displays both the underlying anatomical texture and the high-response areas of interest to the model through a semi-transparent color layer.

[0056] Output a structured diagnostic report. The system encapsulates the predicted category, confidence score, and synthesized heatmap keyframes into a structured data object. This data object is transmitted to the hospital's image storage and transmission system or electronic medical record system through a predefined application programming interface, and displayed on the terminal, containing auxiliary diagnostic information including qualitative conclusions, quantitative grading, and visual images.

[0057] The system performs a confidence-based tiered screening process. Cases are triaged based on generated confidence scores. High and low confidence thresholds are set. Cases with confidence scores above the high confidence threshold are placed in the regular storage queue; cases with confidence scores below the low confidence threshold, or those between the two and with a classification probability difference less than a preset value, are marked as pending verification and added to the pending verification task queue. Simultaneously, a random number generator draws samples from the regular storage queue at a preset sampling rate (e.g., 5%) and adds them to the pending verification task queue.

[0058] An interactive correction interface is provided. The system renders the original image and overlaid heatmap of the case to be verified on the visualization terminal and activates the interactive correction controls. Responding to the physician's operation commands on the interface, the system performs the following actions: receiving instructions to modify the disease category via a drop-down menu; receiving instructions to crop or add areas of interest on the heatmap via a polygonal region selection tool. The system associates and stores the physician-corrected label data, lesion boundary coordinate data, and original image data.

[0059] The system constructs a difficult example database and performs incremental updates. Corrected data is labeled as difficult examples and stored in the incremental training database. This database is independent of the original base training set. The system monitors the number of samples in the incremental training database. When the number of samples reaches a preset batch threshold, or when the system detects idle computing resources, it triggers the model fine-tuning training process.

[0060] Weighted loss optimization is implemented. When fine-tuning training using difficult examples, the system adjusts the calculation logic of the loss function. For samples in the incremental training database, sample weight coefficients are introduced when calculating the cross-entropy loss function. (in The system multiplies the loss value calculated for this sample by... Then, the gradient is calculated during backpropagation. By increasing the proportion of hard examples in the total loss value, the parameter update magnitude for this type of sample is increased, thereby correcting the decision boundary of the model in the feature space.

Claims

1. A multimodal image fusion-based diagnostic and grading system for nasolacrimal duct obstructive diseases, characterized in that, The system includes: The data acquisition and multi-window preprocessing module is configured to acquire computed tomography scan data of the subject's head and perform spatial standardization, and generate bone channel input tensors and soft tissue channel input tensors based on preset dual-window parameters. A dual-stream attention interaction network construction module is configured to construct a dual-stream attention interaction network including a bony feature extraction branch and a soft tissue feature extraction branch; wherein, the bony feature extraction branch and the soft tissue feature extraction branch are respectively used to extract bony topological features and soft tissue pathological features from the bony channel input tensor and the soft tissue channel input tensor, respectively, and use a topology-guided attention unit to generate a spatial attention mask based on the bony topological features, and perform weighted fusion of the soft tissue pathological features to generate a fused feature map; The semi-supervised model training module is configured to construct a hybrid dataset, implement a consistency constraint strategy on unlabeled samples, and update the network parameters of the dual-stream attention interaction network by minimizing a composite loss function that includes supervised loss and unsupervised consistency loss, thereby obtaining the trained dual-stream attention interaction network. The intelligent diagnosis and grading output module is configured to input the bony channel input tensor and the soft tissue channel input tensor of the patient to be diagnosed into the trained dual-stream attention interaction network, and output the disease nature classification and obstruction degree grading results.

2. The multimodal image fusion-based diagnostic and grading system for nasolacrimal duct obstruction disease according to claim 1, characterized in that, The specific methods by which the data acquisition and multi-window preprocessing module generates the bony channel input tensor and the soft tissue channel input tensor include: For the input tensor of the bony channel, the voxel value of the computed tomography data is subtracted from the lower limit of the bone window, then divided by the width of the bone window, and the calculation result is truncated and mapped to the interval of 0 to 1 to suppress soft tissue background and highlight bony anatomical structures; wherein the lower limit of the bone window is the value of the bone window position minus half of the bone window width; For the soft tissue channel input tensor, the voxel value of the computed tomography scan data is subtracted from the lower limit of the soft tissue window, then divided by the soft tissue window width, and the calculation result is truncated and mapped to the interval of 0 to 1 to stretch the grayscale contrast of the lacrimal sac mucosa and the filling material in the lumen; wherein the lower limit of the soft tissue window is the value of the soft tissue window level minus half the soft tissue window width; Finally, the generated bony channel input tensor and soft tissue channel input tensor are combined into parallel input dual-channel data.

3. The multimodal image fusion-based diagnostic and grading system for nasolacrimal duct obstruction disease according to claim 1, characterized in that, The specific steps for generating a spatial attention mask based on the bony topological features using a topology-guided attention unit, and then weighted and fused to produce a fused feature map of the soft tissue pathological features are as follows: The multi-channel bony topological features output by the bony feature extraction branch are compressed into a single-channel feature map using a three-dimensional convolutional layer; The single-channel feature map is mapped to a spatial attention mask with values ​​between 0 and 1 using the Sigmoid activation function. The spatial attention mask represents the probability response intensity of each spatial location belonging to the bony lacrimal duct structure. The spatial attention mask and the soft tissue pathological features output by the soft tissue feature extraction branch are multiplied element-wise to obtain a weighted feature map. The weighted feature map is then joined with the original soft tissue pathological features using an element-wise addition residual join operation to obtain the fused feature map.

4. The multimodal image fusion-based diagnostic and grading system for nasolacrimal duct obstruction disease according to claim 1, characterized in that, The intelligent diagnosis and hierarchical output module also includes a fully connected classification layer, configured as follows: Perform a global average pooling operation on the fused feature map to compress the three-dimensional feature tensor into a one-dimensional feature vector; The one-dimensional feature vector is input into a multilayer perceptron network, which contains two sets of independent output nodes. The first set of output nodes is used to output the classification probabilities of primary blockage and secondary blockage; The second set of output nodes is used to output the graded probabilities of unobstructed, slightly narrow, and severely blocked conditions; The category with the highest probability value is selected as the final structured diagnostic result.

5. The multimodal image fusion-based diagnostic and grading system for nasolacrimal duct obstruction disease according to claim 1, characterized in that, The consistency constraint strategy in the semi-supervised model training module specifically includes: Weak enhancement and strong enhancement operations were applied to the same unlabeled sample, respectively. The weakly enhanced samples are input into the network to obtain the predicted distribution. When the highest confidence of the predicted distribution exceeds a preset threshold, the category corresponding to the predicted distribution is used as a pseudo label. The enhanced samples are input into the network to obtain the enhanced prediction distribution; The constrained network ensures that the predicted distribution of strongly enhanced samples remains consistent with the pseudo-labels. Specifically, the cross-entropy loss between the strongly enhanced predicted distribution and the pseudo-labels is calculated as the unsupervised consistency loss.

6. The multimodal image fusion-based diagnostic and grading system for nasolacrimal duct obstruction disease according to claim 5, characterized in that, The enhanced operation specifically includes: Injecting additive noise that follows a Gaussian distribution into the image simulates quantum noise interference under low-dose scanning conditions; A rectangular region is randomly selected in the three-dimensional space of the image, and the pixel values ​​within the rectangular region are set to zero to simulate local missing or occluded anatomical structures. A non-linear gamma transform is applied to the image pixel values ​​to simulate the contrast differences caused by different reconstruction algorithms.

7. The multimodal image fusion-based diagnostic and grading system for nasolacrimal duct obstruction disease according to claim 1, characterized in that, The semi-supervised model training module is equipped with a dynamic weighting mechanism when calculating the composite loss function: The composite loss function is composed of a weighted sum of the supervised loss of labeled samples and the unsupervised consistency loss of unlabeled samples; A dynamic weight coefficient that gradually increases with the number of training rounds is introduced to adjust the proportion of the unsupervised consistency loss; the unsupervised consistency loss is given a lower weight in the early stage of training and its weight is increased in the later stage of training, and the decision boundary of the model is corrected by utilizing the data distribution characteristics of unlabeled data.

8. The multimodal image fusion-based diagnostic and grading system for nasolacrimal duct obstruction disease according to claim 1, characterized in that, The intelligent diagnosis and grading output module is also equipped with a visualization heatmap generation unit, used for: Extract the spatial attention mask generated by the topology-guided attention unit; The spatial attention mask is upsampled to the same spatial resolution as the original computed tomography data using a bilinear interpolation algorithm. The upsampled mask is mapped as a pseudo-color heatmap, and the pseudo-color heatmap is semi-transparently superimposed on the grayscale image of the soft tissue channel using alpha blending technology to display the internal region of the nasolacrimal duct that the model is interested in.

9. The multimodal image fusion-based diagnostic and grading system for nasolacrimal duct obstruction disease according to claim 1, characterized in that, It also includes an expert collaborative verification mechanism, configured as follows: The diagnostic results are filtered based on the confidence score output by the model, and cases with confidence scores below a preset threshold are marked as cases to be verified. Provides an interactive interface for physicians to correct the diagnostic category of the case to be verified or adjust the area of ​​interest in the heatmap; The cases revised by physicians are stored as difficult case samples in the incremental training database; When fine-tuning the network using the incremental training database, the hard sample is given a higher loss weight than the normal sample.

10. The multimodal image fusion-based diagnostic and grading system for nasolacrimal duct obstruction disease according to claim 1, characterized in that, The spatial standardization processing in the data acquisition and multi-window preprocessing module specifically includes: The head computed tomography data were unified to a standard anatomical space using a rigid registration algorithm, and the line connecting the anterior and posterior commissures was selected as a reference plane for rotational correction. A voxel block containing the bilateral lacrimal duct system was cut out with the midpoint of the line connecting the inner canthi of both sides as the center, and the cut voxel block was resampled to a uniform spatial resolution to eliminate the differences in slice thickness and resolution caused by different acquisition devices.

Images