Pulmonary fibrosis multi-modal staging method based on anatomical priors and clinical conditioning

Abstract

The present application provides a multi-modal staging method for pneumoconiosis based on anatomical priori and clinical conditioning, relating to the field of medical image processing and deep learning. The method encodes multi-dimensional clinical indicators into patient-specific global representation through a clinical semantic encoder, and divides the lung CT image into six anatomical regions through an anatomical priori region attention module. A dynamic index of the guided feature scanning order is generated according to the region attention weight. A state controller is constructed to map the global representation to the evolution parameters of the Mamba state space model, so that the state evolution of the image features is regulated by the clinical image. A clinical conditioned Mamba engine is further designed to perform state space evolution on the CT image features under the guidance of the anatomical scanning order, and the four classification results of pneumoconiosis are output through global aggregation and classification head. The present application can realize the deep fusion of multi-modal information for the automatic and accurate staging of pneumoconiosis, and provide intelligent auxiliary support for clinical evaluation and early intervention.

Description

Technical Field

[0001] This invention relates to the fields of medical artificial intelligence and medical image processing technology, specifically to a multimodal staging method for pneumoconiosis based on anatomical priors and clinical conditions. Background Technology

[0002] Pneumoconiosis is a systemic disease characterized by diffuse pulmonary fibrosis, caused by long-term inhalation of industrial dust during occupational activities, which then remains in the lungs. The disease is highly insidious and irreversible; failure to accurately stage and intervene early poses a serious threat to patients' lives and health, and imposes a heavy economic and medical burden on society and families. With the increasing age of those previously exposed to dust in certain industries and sectors in my country, the incidence and severity of pneumoconiosis remain high. Clinical practice has demonstrated that early and accurate staging of pneumoconiosis is of paramount clinical significance for developing personalized clinical treatment decisions and managing patients' expected health.

[0003] High-resolution chest CT imaging is the primary technique for clinicians to diagnose and assess pneumoconiosis, allowing for artificial staging by observing the spatial distribution of diffuse nodules and the degree of fibrosis in the images. However, the clinical progression of pneumoconiosis is not solely dependent on changes in imaging findings; it is closely related to the patient's individual clinical profile, including demographic indicators (such as age and BMI), laboratory indicators (such as erythrocyte sedimentation rate (ESR) and C-reactive protein (CRP), and key exposure histories and clinical symptoms (such as smoking history, forced vital capacity (FVC), and duration of occupational dust exposure). Traditional fully automated analysis models mostly focus only on imaging features, or simply use direct hard stitching of multimodal features or cross-attention mechanisms for multimodal fusion. This superficial fusion approach easily leads to semantic alignment difficulties between image features and high-dimensional heterogeneous clinical indicator features, readily introducing uncontrollable clinical multimodal noise, and failing to simulate the decision-making logic of clinicians who combine the patient's comprehensive clinical history to guide targeted interpretation of key areas of the imaging. While deep learning networks based on state-space models possess excellent long-sequence modeling capabilities, traditional input-dependent mechanisms lack targeted clinical semantic control when processing multimodal medical data, and fixed raster scanning paths easily overlook the natural anatomical prior topology of the lungs. This invention provides a novel clinically conditionalized anatomical state-space network method that, by fusing complementary features from CT images and heterogeneous clinical indicators, achieves accurate and highly medically interpretable automatic staging of early-stage pneumoconiosis. Summary of the Invention

[0004] This invention aims to address several technical issues in existing auxiliary staging methods for pneumoconiosis. These include difficulties in semantic alignment and the introduction of multimodal noise due to the surface stacking or shallow cross-computation of image features and high-dimensional clinical multimodal indicators; the use of a fixed path in Mamba during image sequence scanning, making it difficult to adaptively fit the irregular spatial geometric distribution of diffuse pneumoconiosis lesions in different pulmonary anatomical regions; insufficient multimodal information fusion, failing to deeply utilize clinical profiles to directly and dynamically adjust the state update rate and evolution rules; and a lack of explicit guidance and screening capabilities for key anatomical lesion areas, making it difficult to simulate the decision-making logic of doctors focusing on important lesion areas and suppressing redundant backgrounds.

[0005] This invention is achieved through the following technical solution: a multimodal staging method for pneumoconiosis based on anatomical prior knowledge and clinical conditions. This method integrates chest CT images with patient clinical profile characteristics to achieve accurate early four-category staging of pneumoconiosis. The specific steps are as follows:

[0006] Step 1: Multimodal data acquisition and dataset construction; Step 2: Preprocessing of CT images and clinical indicators; Step 3: Construction of a clinically conditional anatomical state space network, the network including:

[0007] Step 4: Model training and parameter optimization; Step 5: Model performance evaluation; Step 6: Input the multimodal data to be classified into the trained classification model to obtain the pneumoconiosis staging results.

[0008] Preferably, in step 1, the multimodal data acquisition and dataset construction specifically involve:

[0009] 1.1: Collect chest CT image data and corresponding multidimensional raw clinical indicator data of patients with pneumoconiosis. The raw clinical indicator data includes 50 dimensions of individual patient clinical heterogeneous indicators to ensure the comprehensiveness of the data source.

[0010] 1.2: Based on clinical medical consensus, the 50-dimensional original clinical indicator data is divided into three semantic subspaces, specifically including demographic indicators, laboratory indicators, and exposure history and symptom indicators, thereby establishing a structured grouping that conforms to medical logic at the data level.

[0011] 1.3: Multimodal alignment processing was performed on the acquired imaging data and clinical indicators, and a correspondence was established between the two types of multimodal data and the gold standard label of the four-category staging of patients' actual pneumoconiosis to ensure the accuracy of the monitoring signal;

[0012] 1.4: Based on the aligned multimodal features and stage labels, a research dataset for model training and evaluation is formed.

[0013] Preferably, in step 2, the preprocessing of CT images and clinical indicators specifically includes:

[0014] 2.1: Spatial resampling and size standardization are performed on the acquired chest CT images to meet the spatial geometric dimension requirements of the deep learning network input;

[0015] 2.2: Normalize the grayscale values ​​of the image to eliminate the interference of grayscale differences caused by different CT scanning equipment and scanning parameters on image features, and improve the generalization ability of the algorithm.

[0016] 2.3: The multidimensional original clinical indicators are filled with missing values, standardized and discretized, and then transformed into clinical indicator feature vectors with latent space dimensions aligned and fixed dimensions.

[0017] 2.4: The preprocessed multimodal data is divided into training set, validation set and test set according to a preset ratio to lay the foundation for the subsequent robustness evaluation of the model.

[0018] Preferably, in step 3, constructing the clinically conditioned anatomical state spatial network specifically involves:

[0019] 3.1: Construct an image feature extractor, using a convolutional neural network DenseNet121 to extract dense local two-dimensional spatial feature maps from preprocessed chest CT images, and flatten the two-dimensional spatial feature maps into a one-dimensional image feature sequence along the spatial dimension, which serves as the original input Token for subsequent state space scanning;

[0020] 3.2: Constructing a Clinical Semantic Encoder: The indices of the three semantic subspaces are projected onto a unified latent space through independent, parameter-discretionary three-layer multi-sensor encoders, generating three sets of encoded clinical semantic vectors, each with a dimension of 256. A multi-channel cross-attention matrix is ​​used to calculate the bidirectional semantic correlation between the three sets of vectors, and residual connections containing learnable dynamic gating parameters are introduced to enhance the intensity of key clinical flows. Finally, the three sets of semantic vectors with enhanced interaction are concatenated and input into a gating network, where patient-specific gating weights are generated using a Softmax activation function. Based on the gating weights, the three sets of vectors are weighted and fused to generate a patient-specific global clinical representation vector. This allows for adaptive adjustment of dominant clinical semantics based on individual patient differences (such as elderly or critically ill patients).

[0021] 3.3: Constructing the Anatomical Prior Region Attention Module: The two-dimensional spatial feature map is projected into region attention maps corresponding to the six anatomical regions of the lung using a 1×1 convolution and a softmax activation function; the spatial mean of the attention weights for each region is calculated and sorted in descending order to obtain a region index sequence of length 6; based on the region index sequence and a pre-constructed Token-to-region mapping table, a scan sequence index is generated specifying the order of Tokens in the scanned images. This forces the model to prioritize scanning lung sub-regions with high lesion incidence and key anatomical structures, thus alleviating the problem of tail information forgetting in long sequence scans;

[0022] 3.4: Constructing a State Controller: This includes three independent parameter generators, each generating the global clinical representation vector through two layers of multilayer perceptrons. By performing parameterized nonlinear projection and dimensionality upgrade transformation, three state evolution parameters of the Mamba state space model aligned with the image token dimension are directly generated through inverse mapping, including step size parameter Δ, input control matrix B, and output control matrix C. This completely changes the rigid paradigm of hard stitching of traditional multimodal features, enabling heterogeneous clinical multimodal profiling to directly evolve into a "physical field control signal" that controls the image feature update rate and evolution rules.

[0023] 3.5: Constructing a clinically conditional Mamba state space engine: based on the scan sequence index The one-dimensional image feature sequence is reordered so that the lung regions with high anatomical importance are placed at the beginning of the sequence. The evolution parameters (Δ, B, C) generated by the state controller are used to perform selective state space scanning evolution on the rearranged image feature sequence, so that the CT image features flow in a dedicated clinical control field. Then, the evolved hidden state sequence is generated and reverse sequence recovery is performed to keep it aligned with the original image spatial position to obtain the output sequence.

[0024] 3.6: The output sequence is globally averaged along the Token dimension to obtain a global feature vector, and then processed by multi-task prediction classification head and Softmax normalization to output the four-class classification and staging probability results of pneumoconiosis.

[0025] Preferably, in step 4, model training and parameter optimization specifically involve:

[0026] 4.1: Input the multimodal training data into the constructed clinically conditional anatomical state space network for forward feature training and physical field parameter optimization;

[0027] 4.2: The multi-class cross-entropy loss function is used as the main classification loss function; at the same time, the strongly supervised constraint path is activated during the training phase. The 6 anatomical region feature vectors extracted by the anatomical prior region attention module through weighted mask pooling are input into 6 independent region-level MLP classifiers. The local prediction results are output and the anatomical region supervised loss is calculated. The main classification loss and the anatomical region supervised loss are weighted and summed to force the network to optimize the network parameters under the constraint of clear anatomical boundaries.

[0028] 4.3: The AdamW optimization algorithm combined with a cosine annealing strategy with weight decay is used to dynamically backpropagate and update the weight parameters of the entire multimodal network model.

[0029] 4.4: Iterate through the training by setting an appropriate number of training rounds until the joint total loss function converges to obtain the optimal model parameters.

[0030] Preferably, in step 5, the model performance evaluation specifically includes:

[0031] 5.1: Input multimodal data from the validation or test set into the trained model with locked parameters for phased prediction;

[0032] 5.2: Calculate the overall classification performance index of the four-category classification of pneumoconiosis based on the probability results of the model's forward output, including the four-category accuracy (ACC) and the area under the curve (AUC).

[0033] 5.3: Further calculate the sensitivity, specificity, and F1-score evaluation metrics in a multi-classification environment to conduct an in-depth evaluation of the model's performance at various imbalance levels.

[0034] 5.4: Plot multi-class receiver operating characteristic curves to provide a comprehensive and intuitive quantitative evaluation of the model's overall multimodal decision-making and pneumoconiosis staging performance.

[0035] As a preferred option: In step 6, the multimodal data of the pneumoconiosis patients to be classified are input into the classification model after training, and the jointly established adaptive physical field and dynamic anatomical path are used for joint interactive reasoning to obtain the final accurate staging result of the pneumoconiosis.

[0036] The beneficial effects of this invention are as follows:

[0037] 1. This invention utilizes multidimensional heterogeneous clinical indicator information, employing an explicit clinical semantic encoder to perform subspace partitioning and parallel feature extraction of the indicators. This module divides the indicators into demographic, laboratory indicator, and exposure history groups, aligning with prior clinical knowledge. Furthermore, by introducing a learnable dynamic inter-group gating interaction and aggregation mechanism, it generates patient-specific gating weights. This mechanism allows the model to adaptively adjust the dominant clinical semantics for different patient types (e.g., elderly patients depend on demographic features, and critically ill patients depend on exposure duration), extracting a global clinical profile vector with strong robustness and high medical interpretability, effectively suppressing the multimodal noise contamination problem caused by traditional multimodal hard stitching methods.

[0038] 2. Compared to traditional methods that perform multimodal fusion at the feature level, this invention provides a novel technical implementation path. Through a newly designed parametric projection state controller, it directly inversely maps clinical profile vectors into selective scan state evolution parameters that control the Mamba kernel. This mechanism elevates heterogeneous clinical indicators to "physical field control signals," enabling CT image features to perform state transitions and evolution calculations within a "conditional physical field" entirely defined by the patient's specific clinical characteristics. This deep parametric fusion not only reduces alignment conflicts between heterogeneous modalities in the feature dimension but also injects strong clinical conditional priors into state updates.

[0039] 3. This invention creatively introduces an anatomical prior region attention module, which generates a precise attention map of the six anatomical regions of the lung under strong supervision through 1×1 convolution, and derives a scan sequence index from this map during inference. Compared to the fixed raster spatial scanning order in traditional state-space models, this invention introduces a dynamic scanning mechanism guided by anatomical priors. This forces the Mamba engine to prioritize scanning and calculating lung sub-regions with high lesion incidence or key anatomical priors, significantly mitigating the technical bottlenecks of tail information forgetting and long-range context loss in traditional long-sequence scanning. Without increasing the burden of additional feature tokens, it achieves anatomical alignment between the spatial network and the real anatomical boundary.

[0040] 4. This invention constructs a joint framework comprising a locally strongly supervised path and a forward scan index generation path, optimized using a multi-task joint loss function. During training, Regional CE Loss forces the spatial attention region to adaptively align with the true anatomical boundaries, allowing model parameters to be optimized under anatomical constraints with high objectivity and generalization ability. During inference, the strongly supervised classification path is automatically cut off to maintain the network's lightweight nature and efficient forward computation. Local blocks, adaptive physics field control, and dynamic scan paths are jointly modeled within the Mamba engine, taking into account both the irregular and complex local geometry of diffuse pneumoconiosis lesions and the long-term anatomical context dependence across the entire lung. This enables a more accurate, objective, and interpretable assessment of the staging status of pneumoconiosis patients, providing a highly automated clinical diagnostic auxiliary system for clinical treatment decisions. Attached Figure Description

[0041] Figure 1This is a schematic diagram of the overall structure of the present invention. Detailed Implementation

[0042] The present invention provides Embodiment 1, Embodiment 2 and Embodiment 3.

[0043] Example 1

[0044] Please see Figure 1 This embodiment provides a multimodal staging method for pneumoconiosis based on anatomical priors and clinical conditions, specifically including the following steps:

[0045] Step 1: Multimodal Data Acquisition and Dataset Construction

[0046] First, high-resolution chest CT images and corresponding multidimensional heterogeneous raw clinical indicators were collected from patients with pneumoconiosis. The initial dimension of the raw clinical indicators was 50. Based on prior clinical medical consensus, the 50-dimensional raw clinical indicator vector was divided into three structured semantic subspaces: demographic group, laboratory indicator group, and exposure history and symptom group. The acquired medical data was anonymized and desensitized, and a mapping relationship was established between each patient's multimodal indicators and the actual clinical four-category staging labels of pneumoconiosis, thus forming a research dataset for model training and evaluation.

[0047] Step 2: Preprocessing of CT images and clinical indicators

[0048] Spatial resampling and 3D isotropic alignment were performed on chest CT images. The processed images were then cropped and sized to remove extrapulmonary muscle and air background, preserving intact bilateral lung parenchyma tissue areas. The 2D CT slices were then uniformly cropped to [size missing]. Pixels. Image grayscale values ​​are normalized using a min-max normalization method to linearly map the range of CT values ​​to... Between. Missing values ​​in the original multidimensional clinical indicator data were imputed using the mean, and discrete text indicators were one-hot encoded. Finally, the preprocessed multimodal paired data were sorted according to... The dataset is divided into training set, validation set, and test set.

[0049] Step 3: Constructing a clinically conditioned anatomical state spatial network

[0050] like Figure 1 As shown, an end-to-end differentiable deep learning model is constructed, which specifically includes the following sub-modules:

[0051] 3.1 Image Feature Extractor

[0052] This module is used to extract dense sequences of local spatial features from raw CT images. The input size is... 2D CT images, in which $C$ represents the batch size, and $C$ represents the number of channels. A two-dimensional high-dimensional feature map is extracted using the DenseNet121 backbone feature extraction network while preserving spatial structure information. ,in and In this embodiment, the spatial size after downsampling is set as follows: Then, a serialization operation is performed, flattening the two-dimensional spatial features into a one-dimensional sequence, with each spatial location serving as a token, to obtain the image feature sequence. The number of tokens .

[0053] 3.2 Clinical Semantic Encoder

[0054] Input dimension is The original clinical indicator vector is divided into three semantic sub-vectors: demographic group Laboratory indicator group Exposure history and symptom group Each set of vectors is passed through an independent three-layer MLP encoder with non-shared parameters, projecting the original heterogeneous dimensions into a unified latent space, as shown in the formula:

[0055]

[0056]

[0057]

[0058] Output three sets of encoded clinical semantic vectors, each with a dimension of 1. .

[0059] Subsequently, the three sets of semantic vectors are input into the gating group interaction module. The bidirectional semantic relevance between the three groups is calculated through a multi-channel cross-attention matrix, and learnable dynamic gating parameters are introduced. Controlling the information flow intensity, superimposing residual connections, the interaction formula is:

[0060]

[0061]

[0062]

[0063] Output three sets of semantic vectors with enhanced interaction, maintaining the same dimensionality. .

[0064] Finally, clinical semantic gating aggregation is performed: the three enhanced vectors are concatenated and input into the gating network to generate patient-specific dynamic gating weights. :

[0065]

[0066] The three sets of vectors are weighted and fused based on gating weights to output a global clinical representation vector. Meanwhile, the gating weights are used as an auxiliary output for interpretability analysis.

[0067] 3.3 Anatomical Prior Area Attention Module (APRA Block)

[0068] This module is used to identify the spatial distribution of lung anatomical regions and convert it into a scanning path. Feature maps from the red partition are introduced. Through a Convolutional layers and the Softmax function construct 6-region attention projections, generating attention maps for 6 anatomical regions. These correspond to the upper right lobe, middle right lobe, lower right lobe, upper left lobe, middle left lobe, and lower left lobe, respectively.

[0069] In the forward-pushing path B-sequence generator, the spatial mean of the attention weights for each anatomical region is calculated: Obtain the region weight vector The six regions are sorted in descending order according to their weights to obtain the region priority index sequence. Based on the spatial coordinates of the token in the feature map, a spatial topological mapping table is constructed from the token to the region. Based on region priority and mapping table, the token indexes of the same region are aggregated and flattened to finally generate a complete scan order index:

[0070]

[0071] in, This represents the index lookup function. Functions representing sequence concatenation and flattening. The length is equal to the total number of tokens. Output Used to specify the order of the image sequence scanned by the Mamba engine.

[0072] 3.4 State Controller

[0073] This module is used to visualize global clinical characteristics. The control parameters are transformed into the Mamba state space. The yellow partition contains three parallel, independent parameter generators, each consisting of two layers of MLPs. Three parameter generators are input respectively, and the parameters are directly mapped to the state evolution parameters of the Mamba selective scan through nonlinear projection, as shown in the following formula:

[0074]

[0075]

[0076]

[0077] Directly output step size parameters Input control matrix Output control matrix The output dimension of each parameter matrix is ​​1. The hidden dimension of the evolutionary state Set it to 16.

[0078] 3.5 Clinical Conditioning Mamba State Space Engine

[0079] This is the core of the entire network's fusion and evolution process. First, in the sequence reordering submodule, the input image feature sequence... With Anatomical Scan Index The token sequence is rearranged according to the index, placing regions with high anatomical importance and high attention allocation at the beginning of the sequence. The formula is as follows:

[0080] Output rearranged image feature sequence .

[0081] Subsequently, a clinically conditional selective scan was performed: rearranged features were then analyzed. and clinical parameters Input the Mamba kernel and perform a time-varying state space evolution update. The calculation formula is as follows:

[0082]

[0083]

[0084] Among them, the discretization matrix , ,matrix This is a preset fixed parameter matrix. The output sequence, sorted by anatomical importance, is then calculated from this matrix. .

[0085] Finally, perform sequence recovery: according to The inverse of the sequence restores the original spatial pixel arrangement to maintain topological alignment.

[0086]

[0087] Output spatial alignment sequence .

[0088] 3.6 Global Aggregation and Decision Output

[0089] Space alignment sequence Input the orange partition, perform global average pooling along the token dimension, calculate the full-length mean, and obtain a global feature vector that integrates multimodal features. .Will The input is a multi-task prediction and classification head composed of a multilayer sensing mechanism. The normalization mapping is performed through the Softmax function, and the final output is the staging probability distribution for the four-category classification of pneumoconiosis. Among them, the classification .

[0090] Step 4: Model Training and Parameter Optimization

[0091] The multimodal pairing data from the training set is input into the network constructed above for forward computation. The training process of this invention is jointly guided by two loss functions: the main classification loss uses the cross-entropy loss function. Simultaneously, during the training phase, the strongly supervised local classifier A, which activates the gray partition, is used to enhance the attention map. With feature map Weighted mask pooling is performed to extract feature vectors from the six anatomical regions. Local prediction results are output through six independent region-level multilayer perceptron classifiers. And calculate the supervision loss of the anatomical region. The network's total loss function is constructed by a weighted sum of two parts:

[0092]

[0093] In this embodiment, the balance coefficient The initial learning rate is set to 0.3. The AdamW optimization algorithm is used to update the model parameters through backpropagation, with the initial learning rate set to... The weight decay coefficient is A cosine annealing strategy was used for learning rate scheduling. The training epochs were set to 100, and the batch size was 32. After each epoch, the classification accuracy was evaluated on the validation set, and the optimal weight parameters of the network with the highest accuracy were saved.

[0094] Step 5: Model Performance Evaluation

[0095] The reserved independent test set data is input into the optimal model trained in step 4 for stage prediction. Based on the four-class classification probabilities output by forward propagation, global performance evaluation metrics in a multi-class environment are calculated, specifically including the four-class classification accuracy (ACC), the multi-class confusion matrix, and the area under the curve (AUC) for each stage. To further evaluate the model's discrimination accuracy for imbalanced samples, macroscopic sensitivity, macroscopic specificity, and F1-score evaluation metrics are calculated in detail. Finally, multi-class receiver operating characteristic (ROC) curves are plotted to provide a comprehensive and intuitive quantitative evaluation of the model's overall intelligent multimodal decision-making performance.

[0096] Step 6: Application of pneumoconiosis staging

[0097] Chest CT images of patients with clinically unknown pneumoconiosis to be classified, along with 50 heterogeneous clinical indicators, are preprocessed in step 2 (size registration, missing value imputation, etc.) and then input into the CASS-Mamba staging model, which is constructed in step 3 and trained in step 4. The model performs end-to-end joint interactive inference within an internally adaptively constructed clinical parameter evolution field and anatomical dynamic scanning path, directly outputting the patient's final accurate pneumoconiosis staging result (no pneumoconiosis, stage I, stage II, or stage III), serving as an objective quantitative basis for clinicians to refer to when formulating intervention plans.

[0098] Example 2

[0099] Please refer to it again. Figure 1 This embodiment, based on Embodiment 1, optimizes the loss function and training strategy in the model training and parameter optimization steps to further improve the robustness of the controller when facing high-dimensional parameter mapping. Specifically, in step 4, to constrain overfitting by the three parameter generators in the yellow partition, the total loss function is optimized. Additional low-rank matrix constraint terms were introduced. The weighted penalty term, whose joint total loss function is adjusted as follows:

[0100]

[0101] in, Denotes the Frobenius norm of the matrix, and the penalty coefficient. Set as Simultaneously, a dynamic learning rate linear warm-up strategy was introduced into the optimizer, linearly increasing the learning rate from 0 to 10 over the first 10 epochs. Then, cosine annealing is performed. The number of training rounds is increased to 150 epochs, and an early stopping mechanism is introduced, with a tolerance of 15 epochs. That is, if the accuracy of the validation set fails to improve for 15 consecutive epochs, the training is terminated early. This embodiment significantly improves training stability while suppressing overfitting of the state controller.

[0102] Example 3

[0103] Please refer to Example 3 again. Figure 1 This embodiment, based on Embodiment 1, improves the mapping and projection mechanism of the state controller in the network structure with a low-rank adaptive approach to adapt to small-sample experimental environments and address the performance degradation caused by overfitting. Specifically, in submodule 3.4, the direct brute-force mapping of the high-dimensional time-varying parameter matrix by the two-layer MLP is eliminated. The state controller is instead changed to output only a set of global clinical regulation basis vectors that do not change with the time token. Prior to selective scanning of the green zone, image feature sequences are... The time-varying spatiotemporal scaling matrix is ​​output via a lightweight low-rank linear projection layer. The final evolutionary parameters input to the Mamba engine are generated by conditionally combining the clinical baseline parameters with the image time-varying matrix through an outer product. The calculation formula is as follows:

[0104]

[0105]

[0106]

[0107] This design significantly reduces the number of parameters in the state control partition, uses clinical information as a global macroscopic control physical field, and uses image spatiotemporal features as microscopic local perturbation factors. While maintaining the integrity of the "clinical conditionalization" academic story, it has improved the accuracy of four-class classification by about 6% in small-sample multimodal experiments. The remaining steps are the same as in Example 1.

[0108] Working principle

[0109] Working Principle: The multimodal staging method for pneumoconiosis based on anatomical priors and clinical conditions of this invention works as follows: First, through multimodal input, the system transforms the complex 50-dimensional heterogeneous clinical indicators into a high-purity global clinical representation vector via an explicit semantic encoder. This completely solves the semantic alignment bottleneck of heterogeneous indicators. Next, the system breaks away from the rigid multimodal fusion paradigm of hard-stitching at the feature layer, utilizing a low-rank state controller to inversely map clinical representations into selective scan control parameters in the Mamba state space. At this point, complex clinical information is transformed into "physical field control signals," enabling CT image features to undergo state transitions within a "conditional physical evolution field" entirely defined by the patient's specific clinical characteristics.

[0110] Meanwhile, the anatomical prior region attention module accurately identified six key anatomical regions of the lung on the feature map using a strong supervision mechanism, and converted their spatial attention weights into a scan sequence index by arranging them in descending order. This index offers a different scanning method compared to Mamba's original raster scan path, forcing the engine to prioritize scanning and processing lung sub-regions with high lesion incidence and anatomical criticality, effectively mitigating tail information forgetting and long-term anatomical dependence loss caused by long-sequence scans. Ultimately, guided by the anatomical scan path, the CT image sequence completes in-depth disease evolution within a clinically conditional physical field, and the orange decision head outputs high-precision, self-interpretive four-category staging results, achieving automation and precision in multimodal intelligent diagnosis of pneumoconiosis.

Claims

1. A multimodal staging method for pneumoconiosis based on anatomical priors and clinical conditions, the specific steps of which are as follows: Step 1: Multimodal data acquisition and dataset construction; Step 2: Preprocessing of CT images and clinical indicators; Step 3: Construct a clinically conditioned anatomical state spatial network, the network comprising: Sub-step 3.1: Image feature extractor, which uses a convolutional neural network to extract dense two-dimensional feature maps from the preprocessed chest CT images and flatten them into a one-dimensional image feature sequence; Sub-step 3.2: Clinical semantic encoder, which divides the collected original multidimensional clinical indicators into several groups according to medical semantics, maps them to a unified latent space through independent encoders, and generates patient-specific global clinical representation vectors through gated inter-group interaction and gated aggregation. Sub-step 3.3: Anatomical prior region attention module, generates attention maps of multiple anatomical regions of the lung from the two-dimensional feature map, adopts a region-level dynamic sequence rearrangement method based on anatomical prior, generates a region priority sequence according to the attention weight of each region, and combines it with a pre-calculated Token to region index mapping table to generate a region scan index to guide the subsequent scan order; Sub-step 3.4: State controller, using a parameterized projection method that maps high-dimensional clinical semantics to state space basis vectors, inputs the global clinical representation vectors into three independent parameter generators, mapping them to the state evolution parameters of the Mamba state space model, including step size parameter Δ, input control matrix B, and output control matrix C; Sub-step 3.5: Clinically conditional Mamba engine, rearranges the image feature sequence according to the region scan index, and performs state space evolution on the rearranged image feature sequence using the state evolution parameters (Δ, B, C) to generate the evolved state sequence; Sub-step 3.6: Global aggregation and classification output, global pooling is performed on the evolved state sequence to obtain the global feature vector, and the four-class classification probability of pneumoconiosis is output through the multi-task classification head; Step 4: Model training and parameter optimization; Step 5: Model performance evaluation; Step 6: Input the multimodal data to be classified into the trained classification model to obtain the pneumoconiosis staging results.

2. In step 1 according to claim 1, the multimodal data acquisition and dataset construction specifically comprises: 1.1: Collect chest CT images and corresponding multidimensional raw clinical indicators from patients with pneumoconiosis; 1.2: Based on clinical medical consensus, the collected clinical indicators were classified according to medical semantics into demographic, laboratory test indicators, occupational exposure history, and clinical symptoms; 1.3: The acquired imaging data and clinical indicator data were formatted and aligned, and the correspondence between multimodal data and the actual four-category staging labels of patients with pneumoconiosis was established; 1.4: Based on the aligned labels and data, a research dataset is formed for model training and evaluation.

3. In step 2 of claim 1, the preprocessing of CT images and clinical indicators specifically includes: 2.1: Spatial resampling and size standardization of chest CT images are performed to meet the input size requirements of deep learning networks; 2.2: Normalize the image grayscale values ​​to eliminate the influence of different scanning devices on image features; 2.3: Impute missing values ​​and discretize the multidimensional original clinical indicators to transform them into fixed-dimensional clinical indicator feature vectors; 2.4: Divide the preprocessed multimodal data into training set, validation set and test set according to a preset ratio.

4. In step 3 of claim 1, constructing the clinically conditional anatomical state spatial network specifically involves: 3.1: Construct an image feature extractor, using a convolutional neural network to extract two-dimensional feature maps from preprocessed chest CT images, and flatten the two-dimensional feature maps into a one-dimensional image feature sequence in the spatial dimension; 3.2: Construct a clinical semantic encoder. The clinical indicator feature vectors are divided into three semantic subspaces according to medical semantics: demographic group, laboratory indicator group, and exposure history and symptom group. Each group is projected onto a unified latent space through an independent multilayer perceptron encoder with non-shared parameters, generating three sets of encoded clinical semantic vectors. The bidirectional semantic correlation between the three sets of vectors is calculated through a multi-channel cross-attention matrix, and a learnable dynamic gating parameter is introduced to control the information flow intensity to superimpose residual connections. Subsequently, the three sets of semantic vectors with enhanced interaction are concatenated and input into a gating network. Patient-specific gating weights are generated through a Softmax activation function, and the three sets of vectors are weighted and fused based on the gating weights to generate a patient-specific global clinical representation vector. 3.3: Construct an anatomical prior region attention module. Project the two-dimensional feature map into a region attention map corresponding to the six anatomical regions of the lung through 1×1 convolution and Softmax function. Calculate the spatial mean of the attention weight of each region and sort them in descending order to obtain the region index sequence. Based on the region index sequence and the pre-constructed Token to region mapping table, generate a scan order index that specifies the order of Tokens in the scan image. 3.4: Construct a state controller containing three independent parameter generators, which respectively perform nonlinear projection on the global clinical representation vector through two layers of multilayer perceptrons, directly mapping it to the state evolution parameters of the Mamba state space model. The state evolution parameters include step size parameter Δ, input control matrix B, and output control matrix C. 3.5: Construct a clinically conditional Mamba state space engine, reorder the one-dimensional image feature sequence according to the scan order index, so that the regions with high anatomical importance are placed at the beginning of the sequence, and use the state evolution parameters to perform state space evolution on the rearranged image feature sequence to generate the evolved hidden state sequence and perform sequence recovery to align it with the original image spatial position to obtain the output sequence. 3.6: The output sequence is globally averaged along the Token dimension to obtain a global feature vector, and the four-category staging probability of pneumoconiosis is output through a multi-task classification head.

5. In step 4 according to claim 1, the model training and parameter optimization specifically include: 4.1: Input the multimodal training data into the constructed clinically conditional anatomical state space network for iterative training; 4.2: The cross-entropy loss function is used as the target loss function for model classification; at the same time, the feature vectors of the six anatomical regions are extracted by weighted mask pooling, and the anatomical region supervision loss is calculated by six independent region-level classifiers. The total loss function is constructed by weighted summation of the main classification loss and the anatomical region supervision loss. 4.3: The AdamW optimization algorithm is used to update the model parameters, and the initial learning rate and weight decay coefficient are set. 4.4: Set the number of training rounds to the preset total number of epochs, and obtain the optimal model parameters through backpropagation iteration of the joint loss.

6. In step 5 of claim 1, the model performance evaluation specifically includes: 5.1: Input the multimodal data of the test set into the trained model for staged prediction; 5.2: Calculate the model classification performance metrics based on the prediction results, including the four-class classification accuracy and the area under the curve (AUC). 5.3: Further calculate the multi-class classification sensitivity, specificity, and F1-score evaluation metrics; 5.4: Plot the receiver operating characteristic (ROC) curve to comprehensively evaluate the overall multimodal pneumoconiosis staging performance of the model.

7. In step 6 according to claim 1, the multimodal data to be classified is input into the classification model after training to obtain the pneumoconiosis staging results.

Images