A Multimodal Diagnostic Method and System for Orthopedic Infections Based on Dual-Stream Encoder Fusion

Abstract

This invention discloses a multimodal orthopedic infection diagnosis method and system based on dual-stream encoder fusion. It collects medical image and clinical indicator data, extracts cross-slice features using an image encoder, and generates image embedding representations by adaptive weighted pooling based on information entropy to calculate importance weights. It then uses a clinical encoder to map clinical indicators to generate clinical embedding representations. The system identifies dominant modal patterns, adjusts diagnostic preferences, and generates interactive features. It measures the semantic distance of source features, identifies complementary dimensions, quantifies them into synergy, and generates a joint representation through differential adaptive weight fusion. It assesses confidence levels, selects highly stable features to construct an importance map, and builds decision rules based on high-confidence features to generate preliminary classification results. Based on the preliminary classification results, it identifies difficult samples, extracts boundary feature dimensions to construct boundary pattern quantification sensitivity, classifies risk levels, and constructs a hierarchical diagnostic system to output stratified diagnostic results, achieving deep fusion, accurate classification, and risk stratification of multimodal data.

Description

Technical Field

[0001] This invention relates to the field of intelligent medical imaging diagnostic technology, and in particular to a multimodal orthopedic infection diagnostic method and system based on dual-stream encoder fusion. Background Technology

[0002] Diagnosing orthopedic infections plays a crucial role in the early detection and treatment decisions for diseases such as osteomyelitis and suppurative arthritis. However, a single imaging modality often cannot fully reflect the pathological state of the infection. CT images can show bone destruction and periosteal reaction, but they are difficult to detect early bone marrow edema; MRI can clearly show changes in bone marrow signal and soft tissue abscesses, but it is not as good at detecting subtle cortical bone destruction as CT; laboratory inflammatory markers can quantify the severity of infection, but lack spatial localization information of lesions. This incompleteness of information can easily lead to judgment errors in the diagnosis of early infections and chronic low-grade infections.

[0003] Most existing multimodal data fusion techniques employ feature concatenation or simple weighting, failing to deeply analyze the intrinsic correlations and complementary relationships between different data sources. The fused feature representations suffer from redundant information interference and loss of discriminative information. Furthermore, current methods generally lack an evaluation system for prediction reliability, and lack mechanisms for identifying and handling cases with high diagnostic uncertainty, thus failing to provide differentiated diagnostic recommendations based on the complexity of the case. Summary of the Invention

[0004] This invention discloses a multimodal orthopedic infection diagnosis method and system based on dual-stream encoder fusion. It extracts feature representations from imaging data and clinical data separately through dual-stream encoders, uses an attention mechanism to achieve semantic interaction between modalities, performs adaptive fusion based on modality consistency analysis, extracts reliable features through confidence quantification to construct classification rules and generate preliminary classification results, identifies uncertain samples based on preliminary classification results, performs risk stratification, and constructs a hierarchical diagnosis system, providing an accurate and reliable solution for medical data classification.

[0005] The first aspect of this invention proposes a multimodal orthopedic infection diagnosis method based on dual-stream encoder fusion, comprising the following steps:

[0006] Collect multi-window medical image data and clinical indicator data, generate an image feature matrix from the multi-window medical image data using an image encoder, and generate a clinical feature vector from the clinical indicator data using a clinical encoder.

[0007] The image feature matrix is ​​used to identify cross-slice context features through a global transformer. Slice importance weights are calculated for the cross-slice context features to generate weighted pooling coefficients. Adaptive pooling processing is performed on the cross-slice context features using the weighted pooling coefficients to determine the image embedding representation.

[0008] The clinical feature vector is transformed by a fully connected method to generate a clinical embedding representation. The clinical embedding representation and the image embedding representation are then subjected to bidirectional semantic interaction through a cross-attention mechanism to generate interactive features. The interactive features are then evaluated for modality consistency to quantify modality differences and generate adaptive difference weights. Based on the adaptive difference weights, an adaptive weighted fusion is performed to determine a multimodal joint representation.

[0009] Uncertainty quantization is performed on the multimodal joint representation to generate a confidence distribution. High-confidence features are extracted from the confidence distribution. Classification decision rules are constructed according to the high-confidence features. Preliminary classification results are generated based on the classification decision rules.

[0010] Based on the preliminary classification results, difficult samples are identified, boundary feature patterns are extracted from the difficult samples to generate risk stratification criteria, a hierarchical diagnosis system is constructed based on the risk stratification criteria, and hierarchical diagnosis results are output based on the hierarchical diagnosis system.

[0011] A second aspect of this invention proposes a multimodal orthopedic infection diagnostic system based on dual-stream encoder fusion, comprising:

[0012] The data acquisition module is used to acquire multi-window medical image data and clinical indicator data, generate an image feature matrix from the multi-window medical image data through an image encoder, and generate a clinical feature vector from the clinical indicator data through a clinical encoder.

[0013] The image processing module is used to identify cross-slice context features using the image feature matrix through a global transformer, calculate slice importance weights for the cross-slice context features to generate weighted pooling coefficients, and perform adaptive pooling processing on the cross-slice context features using the weighted pooling coefficients to determine the image embedding representation.

[0014] The fusion processing module is used to perform a fully connected transformation on the clinical feature vector to generate a clinical embedding representation, and to perform bidirectional semantic interaction between the clinical embedding representation and the image embedding representation through a cross-attention mechanism to generate interactive features. The interactive features are then subjected to modality consistency evaluation to quantify modality differences and generate adaptive difference weights. Based on the adaptive difference weights, adaptive weighted fusion is performed to determine a multimodal joint representation.

[0015] The classification decision module is used to perform uncertainty quantification on the multimodal joint representation to generate a confidence distribution, extract high-confidence features from the confidence distribution, construct classification decision rules according to the high-confidence features, and generate preliminary classification results based on the classification decision rules;

[0016] The risk stratification module is used to identify difficult samples based on the preliminary classification results, extract boundary feature patterns from the difficult samples to generate risk stratification criteria, construct a hierarchical diagnosis system based on the risk stratification criteria, and output hierarchical diagnosis results based on the hierarchical diagnosis system.

[0017] The beneficial effects of this invention are reflected in the following points: First, the information value of different slices in three-dimensional medical images varies greatly, and the contribution of key slices containing lesions and background slices to diagnosis is not the same. To address this characteristic, a window comparison test is used to locate lesion clusters and apply weighted enhancement, avoiding background interference caused by equal-weighting. Furthermore, an adaptive weighted pooling strategy based on information entropy is introduced, dynamically allocating aggregation weights according to the information content of each slice. Information-rich lesion slices receive higher feature contributions, while the influence of information-sparse edge slices is suppressed, improving the discriminative ability of image feature representation. Simultaneously, discrete clinical indicators are mapped to a unified high-dimensional semantic space, establishing a feature alignment foundation for the effective fusion of heterogeneous modalities. Second, traditional feature stitching or simple weighting methods fail to deeply analyze the semantic associations and complementary characteristics between modalities. This invention overcomes this limitation by employing a cross-attention mechanism to establish a bidirectional semantic association between image and clinical modalities, identifying the dominant relationship and collaborative patterns of the two modalities across different feature dimensions. More importantly, to address the differences in synergy between different modalities across complementary dimensions, a difference-adaptive weight is generated for fine-grained fusion adjustment: dimensions with high synergy fully integrate information from both modalities, while dimensions with low synergy avoid feature conflicts. This difference-aware adaptive fusion strategy effectively solves the problems of information redundancy and loss of discriminative information in heterogeneous modality fusion, improving the discriminative power of the fused features. Finally, considering the varying diagnostic difficulties of different cases in clinical practice, a single classification model is insufficient to adapt to all scenarios. Therefore, a confidence quantification mechanism is introduced to evaluate the reliability of the fused features. Stable and reliable high-confidence features are selected through perturbation resistance testing, and interpretable decision rules are constructed based on these features to generate preliminary classification results, transforming the classification process into a transparent rule system. Further analysis of the preliminary classification results identified difficult samples with high prediction uncertainty. Boundary feature analysis was performed on these difficult samples, and the samples were divided into different risk levels by quantifying boundary sensitivity. A hierarchical diagnostic system was constructed to provide differentiated diagnostic strategies for cases of different risk levels: low-risk cases achieved rapid automatic diagnosis to improve efficiency, while high-risk cases triggered expert intervention to ensure accuracy. This achieved a reasonable match between medical resources and case complexity, and improved the reliability and practicality of diagnosis. Attached Figure Description

[0018] The accompanying drawings illustrate specific examples of the technical solutions described in this invention and, together with the detailed embodiments, form part of the specification, serving to explain the technical solutions, principles, and effects of this invention.

[0019] Unless otherwise specified or defined, the same reference numerals in different figures represent the same or similar technical features, and different reference numerals may be used to represent the same or similar technical features.

[0020] Figure 1 This is a flowchart illustrating the multimodal orthopedic infection diagnosis method based on dual-stream encoder fusion according to the present invention.

[0021] Figure 2 This is a structural block diagram of the multimodal orthopedic infection diagnostic system based on dual-stream encoder fusion according to the present invention. Detailed Implementation

[0022] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0023] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0024] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0025] The technical solutions of the embodiments of this application will be described below.

[0026] like Figure 1 As shown, this embodiment of the invention provides a multimodal orthopedic infection diagnosis method based on dual-stream encoder fusion, including the following steps S110-S150:

[0027] Step S110: Collect multi-window medical image data and clinical indicator data, generate an image feature matrix from the multi-window medical image data using an image encoder, and generate a clinical feature vector from the clinical indicator data using a clinical encoder.

[0028] Specifically, multi-window medical imaging data is acquired from medical imaging equipment. For CT scans, multi-window imaging data from different anatomical planes are acquired, including axial and coronal slices. Each anatomical plane contains two window settings: bone window and soft tissue window. Bone windows are used to display bone destruction, periosteal reaction, and sequestrum formation, while soft tissue windows are used to display soft tissue swelling, abscess formation, and interfascial effusion. Axial slices display transverse bone structure, while coronal slices facilitate assessment of the longitudinal extent of lesions. For MRI scans, three imaging sequences are acquired: T1-weighted sequences, T2-weighted sequences, and STIR (Short Time Inversion Recovery) sequences. T1 sequences emphasize anatomical structures, T2 sequences highlight bone marrow edema and soft tissue inflammation areas, and STIR sequences suppress fat signals to enhance the contrast of inflammatory lesions. The imaging data in each window contains multiple consecutive slices, typically 30-200 slices. The images in each window correspond to the same anatomical location of the same patient and have spatial registration relationships.

[0029] In some embodiments, generating an image feature matrix from the multi-window medical image data via an image encoder includes: splitting the multi-window medical image data into image sequences of individual windows; applying a shared weight convolutional network to each slice of the image sequence of individual windows to extract local spatial features; performing inter-window comparison tests on the local spatial features to identify areas of difference clusters and label them as key areas of interest; and combining the local spatial features and the key areas of interest to form an image feature matrix.

[0030] Multi-window medical image data is split into individual window image sequences. The multi-window medical image data is split into windows, separating image data from different anatomical planes, window settings, or imaging sequences into independent window image sequences. For CT images, four window image sequences are separated from the multi-window medical image data: axial bone window image sequence, axial soft tissue window image sequence, coronal bone window image sequence, and coronal soft tissue window image sequence. Each sequence contains all slices at the same anatomical location. The axial bone window image sequence contains N slices, each slice being 512×512 pixels in size. The other three window image sequences also contain N slices, with consistent slice size and number of slices. For MRI images, three window image sequences are separated from the multi-window medical image data: T1 sequence image, T2 sequence image, and STIR sequence image. Slices in each window image sequence are arranged in anatomical order, numbered sequentially from head to foot or from top to bottom. The spatial correspondence of each window image sequence is verified to ensure that the i-th slice of different windows corresponds to the same anatomical plane. The split window image sequences are input into the subsequent processing flow separately to maintain the independence of each sequence.

[0031] Local spatial features are extracted by applying a shared-weight convolutional network slice by slice to each window image sequence. Each slice in each window image sequence is input into the shared-weight convolutional network. The shared-weight convolutional network uses a ResNet-50 architecture, receives 512×512 pixel single-channel grayscale image slices, and outputs a 16×16×2048 feature map. All window image sequences share the same set of convolutional network parameters; that is, axial bone window images, axial soft tissue window images, coronal bone window images, and coronal soft tissue window images use the same convolutional kernel weights. T1, T2, and STIR sequences also use the same convolutional kernel weights. The convolutional network is applied to the i-th slice of each window image sequence to obtain the feature map F_i of that slice, with a size of 16×16×2048. All slices of each window image sequence are sequentially input into the convolutional network to extract the local spatial features of each slice. For a window image sequence containing N slices, N feature maps are obtained. Each feature map contains 16×16 spatial locations, and each location corresponds to a 2048-dimensional feature vector. The local spatial features extracted from each window image sequence maintain spatial correspondence, and the local spatial features of the i-th slice in different windows correspond to the same anatomical location.

[0032] To identify and highlight areas of high interest, inter-window comparison tests are performed to analyze local spatial features. For the i-th slice at the same anatomical level, local spatial features of each window are extracted. Feature differences between windows are calculated; for each spatial location on the feature map, the difference between feature vectors of different windows is calculated. A larger difference indicates a greater difference in tissue characteristics displayed by different windows at that location. For the four windows in CT, feature differences between bone and soft tissue windows within the same anatomical plane, as well as feature differences between different anatomical planes, are calculated and summarized into a comprehensive difference map. For the three windows in MRI, feature differences between each pair of windows are calculated and summarized into a comprehensive difference map. Spatial clustering analysis is performed on the difference maps to identify spatial regions with significant feature differences. Locations with difference values ​​exceeding a set threshold are marked as high-difference points. Connectivity analysis is performed on these high-difference points, grouping spatially adjacent high-difference points into the same cluster. These clusters are marked as areas of high interest, and their spatial coordinates are recorded. These clusters typically correspond to lesion locations or areas of tissue abnormality. Different windows exhibit significant differences in imaging characteristics at lesion locations, thus showing substantial differences in local spatial features.

[0033] A feature matrix is ​​formed by aggregating local spatial features and key areas of interest. For an image sequence containing N slices, the local spatial features of all slices are aggregated to form a three-dimensional feature tensor with dimensions N×16×16×2048. For spatial locations with key areas of interest, the feature vectors of those locations are weighted and enhanced. A location weight vector is set: a weight of 2.0 for locations within the set of key areas of interest, and a weight of 1.0 for other locations. The location weights are applied to the local spatial features to obtain weighted features. The weighted features of multiple windows are stitched together along the channel dimension. For CT images, bone window and soft tissue window features from the same anatomical plane are first fused, and then the fused features from the axial and coronal planes are stitched together, resulting in a feature dimension of N×16×16×4096. For MRI images, the weighted features of T1, T2, and STIR sequences are stitched together along the channel dimension, resulting in a feature dimension of N×16×16×6144. Global average pooling is performed on the stitched features, averaging them across the spatial dimension to obtain the aggregated feature vector for each slice. The dimension for CT images is N×4096, and for MRI images it is N×6144. The aggregated feature vectors of the N slices are arranged layer by layer to form the image feature matrix M_img. The matrix size for CT images is N×4096, and for MRI images it is N×6144. Each row in the image feature matrix corresponds to the feature representation of one slice, and each column corresponds to a feature dimension. The image feature matrix integrates the local spatial features of multi-window image sequences and highlights the feature expression of lesions or abnormal areas through weighted enhancement of key focus locations.

[0034] Clinical indicator data is processed by a clinical encoder to generate clinical feature vectors. Continuous indicators in the clinical indicator data are normalized, scaling indicators such as age, height, and weight to the range of 0-1. Categorical indicators in the clinical indicator data are one-hot encoded, with gender encoded as a 2-dimensional vector: male [1,0] and female [0,1]. Laboratory test results are standardized by subtracting the mean from each indicator and dividing by the standard deviation, resulting in a data distribution with a mean of 0 and a standard deviation of 1. The preprocessed clinical indicator data are concatenated into a clinical indicator vector with dimension M, where M is the total number of clinical indicators. This clinical indicator vector is then input into the clinical encoder. The clinical encoder employs a fully connected neural network structure, containing three fully connected layers. The first fully connected layer maps the M-dimensional input to 256 dimensions, using ReLU activation. The second fully connected layer maps the 256 dimensions to 128 dimensions, also using ReLU activation. The third fully connected layer maps the 128 dimensions to 64 dimensions, serving as the final clinical feature vector output. Dropout layers were added between each fully connected layer with a dropout rate of 0.3 to prevent overfitting. The clinical feature vector, denoted as V_clinical, has a dimension of 64. This vector integrates the patient's basic information, inflammatory marker test results, and other clinical indicators, providing a feature representation of the patient's overall infection status and the severity of inflammation.

[0035] Step S120: Using the image feature matrix, identify cross-slice context features through a global transformer, calculate slice importance weights for the cross-slice context features to generate weighted pooling coefficients, and use the weighted pooling coefficients to perform adaptive pooling processing on the cross-slice context features to determine the image embedding representation.

[0036] Specifically, cross-slice context features are identified using an image feature matrix through a global transformer. The image feature matrix has a size of N×D, where N is the number of slices and D is the feature dimension of each slice; for CT images, D=4096, and for MRI images, D=6144. The global transformer employs a Transformer architecture, using a multi-head self-attention mechanism to calculate feature associations between slices and capture long-range dependencies. Each row of the image feature matrix is ​​treated as an independent slice feature vector, and positional encoding is added to the feature matrix. After multi-layer encoders, cross-slice context features with N×D dimension, incorporating global context information, are obtained. For orthopedic CT images, cross-slice context features can capture morphological changes of infected lesions in adjacent layers, the longitudinal extent of bone destruction, and the spatial relationship between the lesion and surrounding soft tissues, periosteum, and joint cavity.

[0037] In some embodiments, the step of calculating slice importance weights and generating weighted pooling coefficients based on the cross-slice context features includes: measuring the information entropy value of each slice for the cross-slice context features; sorting and identifying high-information-content slice positions and low-information-content slice positions according to the information entropy values; calculating slice importance weights according to the high-information-content slice positions to establish an attention weight mapping; and using the attention weight mapping in combination with the low-information-content slice positions to generate weighted pooling coefficients.

[0038] The information entropy value of each slice is measured for cross-slice context features. The information entropy value of each slice is calculated for cross-slice context features to quantify the information content of each slice. Cross-slice context features contain N slices, each corresponding to a D-dimensional feature vector. For the feature vector of the i-th slice, the information entropy of that vector is calculated. Information entropy reflects the uncertainty and information content of the feature vector; a higher entropy value indicates a more dispersed feature distribution and greater information content, while a lower entropy value indicates a more concentrated feature distribution and less information content. The feature vectors are normalized to convert the feature values ​​into a probability distribution. The information entropy value is calculated based on the probability distribution, using the formula H_i = -Σp_i(k) × log(p_i(k)), where H_i is the information entropy value of the i-th slice, p_i(k) is the probability value of the k-th feature dimension, and log is the natural logarithm. The information entropy values ​​are calculated for each of the N slices, resulting in an information entropy value sequence. Sections containing infected lesions or bone destruction typically have higher information entropy values ​​because the characteristic distribution of the lesion area is more complex and diverse, with bone destruction boundaries, periosteal reaction, and soft tissue swelling creating complex radiographic features. Sections of normal bone tissue areas have relatively lower information entropy values, and their characteristic distribution is more uniform and uniform. For patients with osteomyelitis, the information entropy value of sections containing infected lesions is usually 15-30% higher than that of normal bone tissue sections.

[0039] High-information-content slice locations are identified by sorting according to their entropy values. The entropy value sequence is sorted in descending order to obtain a sorted entropy value sequence and corresponding slice location indices. In the sorted sequence, the first part corresponds to slices with higher entropy values, and the latter part corresponds to slices with lower entropy values. The K slices with the highest entropy values ​​are selected as high-information-content slices; the K value is dynamically determined based on the total number of slices N, and the top 30% of slices are selected. For an image sequence containing 50 slices, the 15 slices with the highest entropy values ​​are selected as high-information-content slices. A set of high-information-content slice location indices is recorded; these locations typically correspond to slices containing lesions, tissue boundaries, or complex anatomical structures. The M slices with the lowest entropy values ​​are selected as low-information-content slices; the M value is also selected from the bottom 30% of slices. A set of low-information-content slice location indices is recorded; these locations typically correspond to normal bone tissue, periarticular soft tissue, or slices with low information content. For orthopedic CT images, low-information-content slices may be located in the distal bone shaft or the edge region of the joint space, areas lacking obvious signs of infection.

[0040] An attention weight mapping is established by calculating the importance weights of slices based on their high-information-content slice locations. This mapping maps slice locations to corresponding weight values, reflecting the slice's contribution to the final diagnostic result. Slices in high-information-content slice locations are assigned higher base weights, set to 2.0. Other slices have a base weight of 1.0, maintaining standard weights. Slices in low-information-content slice locations have a base weight of 0.5 to reduce their impact on feature aggregation. Based on the base weights, fine-tuning is performed using information entropy values. For the i-th slice, its information entropy value is normalized to the 0-1 range to obtain η_i. Then, the fine-tuning weight w(i) = w_base(i) × (1 + 0.5 × η_i) is calculated, where w(i) is the fine-tuning weight of the i-th slice, and w_base(i) is the base weight. The fine-tuning weights are finely adjusted based on the entropy value, with slices having higher entropy values ​​receiving additional weight gains. The weights of all slices are normalized to ensure that the sum of the weights is 1. The normalized weights constitute the attention weight map. The attention weight map reflects the relative importance of each slice in the image embedding representation generation process. The weight of high-information slices is usually 3-4 times that of low-information slices.

[0041] Weighted pooling coefficients are generated using an attention weight mapping combined with the location of low-information slices. The weight values ​​in the attention weight mapping are extracted, and the weight vector is directly used as the initial value for the weighted pooling coefficients. For slices at low-information slice locations, the weighted pooling coefficients are further adjusted. It is determined whether the low-information slice is at the edge of the image sequence, defined as a slice in the first or last 5 layers. If the low-information slice is at the edge, its weighted pooling coefficient is reduced to 50% of its original value to reduce interference from the edge background region on feature aggregation. If the low-information slice is in the middle of the sequence, its weighted pooling coefficient remains unchanged. The adjusted weighted pooling coefficients are then renormalized to ensure that the sum of the coefficients is 1. The final weighted pooling coefficients reflect the actual contribution ratio of each slice in feature aggregation; the coefficient values ​​are larger for high-information slices, and smaller for low-information slices, especially those at the edges. For an orthopedic CT image containing 50 slices, the sum of the weighted pooling coefficients of 5-8 slices containing infected lesions (bone destruction or soft tissue abscess) may reach 0.4-0.5, while the sum of the coefficients of the 10 slices of normal bone tissue or periarticular soft tissue at the periphery is only 0.1-0.15.

[0042] An adaptive pooling process using weighted pooling coefficients is employed to determine the image embedding representation by performing cross-slice context features. The cross-slice context features have a dimension of N×D, and the weighted pooling coefficient vector has a dimension of N×1. For each feature dimension of the cross-slice context features, a weighted average is calculated. For the k-th feature dimension, the weighted average is calculated as E_k=Σ(β_i×F_i(k)), where E_k is the value of the k-th feature dimension after aggregation, β_i is the weighted pooling coefficient of the i-th slice, and F_i(k) represents the value of the i-th slice in the k-th feature dimension, with the summation ranging from i=1 to N. Weighted averages are calculated for each of the D feature dimensions to obtain the aggregated feature vector, which has a dimension of 1×D. The aggregated feature vector is the image embedding representation. The image embedding representation integrates the global feature information of N slices. By adjusting the weighted pooling coefficients, it emphasizes the feature contribution of high-information slices and suppresses the interference from low-information slices and edge regions. Image embeddings capture the three-dimensional morphological features of infected lesions, the spatial distribution patterns of bone destruction and soft tissue inflammation, and the relationship between the lesions and surrounding bone structures. L2 normalization is applied to the image embeddings, normalizing the magnitude of the feature vectors to 1. This normalized image embedding facilitates fusion with clinical feature vectors and facilitates feature similarity comparisons between different patients. The image embeddings are recorded as the final representation of the image features for subsequent infection diagnosis, severity assessment, or prognostic prediction tasks. For orthopedic infection diagnosis, image embeddings provide a global feature description of the lesions, and combining them with inflammatory markers can improve diagnostic accuracy.

[0043] Step S130: Perform a fully connected transformation on the clinical feature vector to generate a clinical embedding representation. Use a cross-attention mechanism to enable bidirectional semantic interaction between the clinical embedding representation and the image embedding representation to generate interactive features. Perform modality consistency assessment on the interactive features to quantify modality differences and generate differential adaptive weights. Perform adaptive weighted fusion based on the differential adaptive weights to determine the multimodal joint representation.

[0044] Specifically, a fully connected transformation is performed on the clinical feature vectors to generate a clinical embedding representation. The clinical feature vectors have a dimension of 64, while the image embedding representation has a dimension of D, where D=4096 for CT images and D=6144 for MRI images. Due to the significant difference in dimensions between the clinical feature vectors and the image embedding representation, the clinical feature vectors need to be projected into the same high-dimensional space as the image embedding representation to achieve effective fusion of the two modalities in a unified feature space. A fully connected transformation network is constructed to perform nonlinear mapping on the clinical feature vectors. The fully connected network contains multiple hidden layers and uses the ReLU activation function to introduce nonlinear transformation capabilities. Through multi-layer mapping, the 64-dimensional clinical feature vectors are gradually expanded to a D-dimensional high-dimensional representation space to obtain the clinical embedding representation. The dimension of the clinical embedding representation is 1×D, the same as the dimension of the image embedding representation. During the mapping process, the fully connected network learns the potential correlation patterns between clinical indicators, encoding discrete clinical information such as patient laboratory test results and demographic characteristics into continuous high-dimensional vector representations. Clinical embeddings preserve the semantic relationships between clinical indicators, such as the synergistic changes in body temperature and white blood cell count, and the correlation between CRP levels and symptom duration, while also possessing the ability to interact with imaging features in a high-dimensional space. For patients with orthopedic infections, the clinical embeddings integrate infection-related clinical factors such as body temperature, white blood cell count, CRP, ESR, PCT, symptom duration, and previous surgical history, forming a comprehensive infection risk feature vector in a high-dimensional space.

[0045] In some embodiments, the step of generating interactive features by enabling bidirectional semantic interaction between the clinical embedding representation and the image embedding representation through a cross-attention mechanism includes: retrieving the image embedding representation using the clinical embedding representation to generate a clinically guided attention map; retrieving the clinical embedding representation using the image embedding representation to generate an image-guided attention map; comparing the clinically guided attention map and the image-guided attention map to identify modality asymmetry dominant patterns and generate dominant transition features; and adjusting diagnostic preferences based on the dominant transition features to generate interactive features.

[0046] Clinical embeddings are used to retrieve image embeddings, generating a clinically-guided attention map. Both the clinical and image embeddings are 1×D in dimension. The clinical embedding is used as the query vector, and the image embedding as the key and value vectors. A similarity score is calculated between the clinical and image embeddings using the following formula: Where S_c2i is the similarity score from clinical to image, Q_c is the clinical embedding representation as the query vector, Ki is the image embedding representation as the key vector, and d is the feature dimension. To prevent excessively large dot product values ​​from being skewed by scaling factors, the similarity scores are normalized using the softmax function to obtain clinically-oriented attention weights. These normalized weights range from 0 to 1 and sum to 1. The clinically-oriented attention weights are then multiplied by the image embedding representation as a value vector to obtain the image feature representation retrieved from clinical information. The clinically-oriented attention map reflects the pattern of clinical information's focus on image features; image feature dimensions with high attention weights represent image manifestations highly correlated with clinical features. For patients with high fever and significantly elevated white blood cell counts, the clinically-oriented attention map tends to focus on image features related to acute infection, such as bone destruction, periosteal reaction, and soft tissue swelling. The clinically-oriented attention map has a 1×D dimension, where the value of each dimension represents the intensity of activation of that image feature dimension by clinical information.

[0047] Image-guided attention maps are generated by retrieving clinical embeddings using image embeddings. The image embeddings are used as the query vector, and the clinical embeddings as the key and value vectors. A similarity score is calculated between the image and clinical embeddings using the following formula: Where S_i2c is the image-to-clinical similarity score, Qi is the image embedding representation as the query vector, K_c is the clinical embedding representation as the key vector, and d is the feature dimension. To prevent excessively large dot product values, a softmax function is applied to normalize the similarity scores, yielding image-guided attention weights. These weights are then multiplied by the clinical embedding representation as a value vector to obtain the clinical feature representation retrieved from the image information. The image-guided attention map reflects the attention pattern of image information towards clinical features; clinical feature dimensions with high attention weights represent clinical indicators highly correlated with the imaging findings. For patients showing significant bone destruction and soft tissue abscesses on imaging, the image-guided attention map tends to focus on clinical features related to acute infection, such as significantly elevated CRP, leukocytosis, and fever. The image-guided attention map has a 1×D dimension, where the value of each dimension represents the intensity of activation of that clinical feature dimension by the image information.

[0048] For example, the step of comparing the clinically guided attention map and the image-guided attention map to identify modal asymmetric dominant modes and generate dominant transition features includes: calculating the difference in response intensity between the clinically guided attention map and the image-guided attention map to generate an asymmetric response distribution; identifying the dominant modality region from the asymmetric response distribution; performing influence propagation analysis on the dominant modality region to construct a modality dominance topology; and extracting cross-modal transmission paths based on the modality dominance topology to generate dominant transition features.

[0049] The asymmetric response distribution is generated by calculating the response intensity differences between clinically guided attention maps and image-guided attention maps. Both clinically guided and image-guided attention maps are 1×D in dimension. The response intensity difference is calculated for each feature dimension of the two attention maps, using the absolute value difference to measure the degree of asymmetry between the two modalities in that dimension. The response intensity difference reflects the degree of asymmetry between the two modalities in that feature dimension; a larger difference indicates that one modality's attention to that dimension is significantly stronger than the other, while a smaller difference indicates that the two modalities' attention to that dimension is comparable. Statistical analysis is performed on the response intensity differences across the D feature dimensions, calculating the mean and standard deviation of the differences. The response intensity differences are normalized, scaling the difference values ​​to the range of 0-1. The normalized differences constitute the asymmetric response distribution. The asymmetric response distribution is 1×D in dimension, with larger values ​​corresponding to feature dimensions with stronger modality asymmetry. The spatial pattern of the asymmetric response distribution is analyzed to identify the feature dimension intervals where differences are concentrated. For the task of diagnosing orthopedic infections, the asymmetric response distribution showed that the imaging modality was dominant in the feature dimensions related to bone destruction morphology and periosteal reaction type, while the clinical modality was dominant in the feature dimensions related to body temperature and inflammatory markers.

[0050] Identify dominant mode regions from asymmetric response distributions. The asymmetric response distribution has a dimension of 1×D, and its value reflects the intensity of modal asymmetry. A dominance threshold is set, calculated using the following formula: Where T_dom is the dominant threshold, The mean of the asymmetric response distribution. The standard deviation of the asymmetric response distribution is used as a threshold to filter significantly asymmetric feature dimensions. Feature dimensions with values ​​greater than the dominant threshold in the asymmetric response distribution are marked as strongly asymmetric dimensions. Modality assignment is determined for strongly asymmetric dimensions by comparing the original values ​​of the clinical-guided attention map and the image-guided attention map for that dimension. If the value of the clinical-guided attention map is larger, the dimension is assigned to the clinically dominant region; if the value of the image-guided attention map is larger, the dimension is assigned to the image-guided dominant region. The feature dimension index sets of the clinically dominant region and the image-guided dominant region are recorded. For a representation space containing D feature dimensions, the clinically dominant region may contain 20-30% of the feature dimensions, and the image-guided dominant region may contain 25-35% of the feature dimensions. The semantic meaning of the dominant modality regions is analyzed. The clinically dominant region typically corresponds to a semantic space related to inflammatory markers, body temperature, and medical history characteristics, while the image-guided dominant region typically corresponds to a semantic space related to bone destruction morphology, periosteal reaction type, and soft tissue swelling degree.

[0051] A modality dominance topology is constructed by performing influence propagation analysis on the dominant modality region. The dominant modality regions include clinically dominant and imaging-dominant regions, each containing several feature dimensions. The influence propagation paths of the dominant modality region on other feature dimensions are analyzed. Feature correlations between feature dimensions within the dominant region and feature dimensions in non-dominant regions are calculated, and cosine similarity is used to measure the degree of correlation between feature dimensions. For the i-th feature dimension of the clinically dominant region, its cosine similarity with all feature dimensions in non-dominant regions is calculated. An influence propagation threshold of 0.5 is set; an influence propagation path is considered to exist when the similarity is greater than 0.5. A modality dominance topology graph is constructed, where nodes represent feature dimensions, edges represent influence propagation paths, and the edge weights are the similarity values. The modality dominance topology graph reflects how the dominant modality region influences other regions through feature associations; clinically dominant feature dimensions may propagate their influence to the imaging-dominant region through high correlation. The connectivity and central nodes of the topology graph are analyzed; central nodes correspond to feature dimensions with strong influence. For the diagnosis of orthopedic infections, the clinical CRP level characteristics may be strongly correlated with the periosteal reaction characteristics in imaging, forming a cross-modal influence propagation path.

[0052] Dominant transition features are generated by extracting cross-modal transmission paths based on modal dominance topology. The modal dominance topology contains nodes and edges; nodes represent feature dimensions, and edges represent influence propagation paths. Transmission paths from the clinically dominant region to the imaging-dominant region, and from the imaging-dominant region to the clinically dominant region, are identified in the topology graph. Cross-modal transmission paths connecting different dominant regions are identified, and the key paths with the highest influence are selected. Dominant transition features are generated based on these key transmission paths. For each feature dimension on a path, a transition weight is assigned to that dimension based on its position within the path and its influence along the path. Feature dimensions at the path's starting point receive higher source modality weights, feature dimensions at the path's ending point receive higher target modality weights, and feature dimensions in the middle of the path receive a mixed weight of both modalities. The transition weights of all key transmission paths are aggregated to obtain the final transition weight for each feature dimension. The dominant transition features are then weighted using these transition weights after cross-attention, with a dimension of 1×D.

[0053] Interactive features are generated based on diagnostic preference adjustment using dominant transformation features. Dominant transformation features reflect the dominance relationship between the clinical and imaging modalities across different feature dimensions. Diagnostic preference adjustment is applied to the dominant transformation features, adjusting the modality fusion strategy according to the characteristics of the specific diagnostic task. For diagnostic tasks that rely on morphological features, such as assessing bone destruction in chronic osteomyelitis, the feature weights of the imaging-dominant region are increased. For diagnostic tasks that require laboratory testing, such as determining the severity of acute infections, the feature weights of the clinical-dominant region are increased. A diagnostic preference parameter λ is set, ranging from 0 to 1. λ=0.5 indicates balanced fusion of the two modalities; λ>0.5 favors the imaging modality, and λ<0.5 favors the clinical modality. The dominant transformation features are adjusted according to the diagnostic preference parameter. The adjusted interactive feature is calculated using the formula F_inter=λ×F_dom_image+(1-λ)×F_dom_clinical, where F_inter is the interactive feature, λ is the diagnostic preference parameter, F_dom_image is the feature of the imaging-dominant part, and F_dom_clinical is the feature of the clinical-dominant part. For the diagnostic task of chronic osteomyelitis, λ=0.6 was set, placing greater emphasis on radiographic bone destruction features. For the task of assessing the severity of acute infection, λ=0.4 was set, placing greater emphasis on inflammatory markers such as CRP and white blood cell count. The interactive feature has a dimension of 1×D, which integrates semantic information from clinical and radiographic embedding representations. It achieves bidirectional semantic interaction through a cross-attention mechanism and is adaptively adjusted according to modality dominance and diagnostic preferences.

[0054] In some embodiments, the step of performing modal consistency assessment on the interaction features to quantify modal differences and generate adaptive weights includes: separating image source features and clinical source features for the interaction features; performing semantic distance measurement on the image source features and the clinical source features to generate a modal complementarity gap dimension; performing gap filling analysis along the modal complementarity gap dimension to determine the complementary enhancement type; and quantifying the complementary enhancement type into modal synergy and mapping it to generate adaptive weights.

[0055] We separate image source features and clinical source features for interactive features. The interactive features are 1×D in dimension and are the result of the interaction between the clinical embedding representation and the image embedding representation through a cross-attention mechanism, containing mixed information from both modalities. We use feature decomposition to separate the interactive features into image source features and clinical source features. We construct two projection matrices, the image projection matrix and the clinical projection matrix, to capture the feature patterns of the two modalities, respectively. We multiply the interactive features by the image projection matrix and the clinical projection matrix, respectively, to obtain the image source features and clinical source features. The image source features reflect the portion of the interactive features aligned with the image embedding representation, and the clinical source features reflect the portion of the interactive features aligned with the clinical embedding representation. We normalize the image source features and clinical source features to ensure consistent numerical scales. We analyze the distribution characteristics of the image source features and clinical source features, calculating their respective means and variances.

[0056] Semantic distance measurement is performed on image source features and clinical source features to generate modal complementarity gap dimensions. Both image source features and clinical source features have a dimension of 1×D. The semantic distance between the two features is calculated using the formula D_semantic = ||F_image - F_clinical||_2, where D_semantic is the semantic distance, F_image is the image source feature, F_clinical is the clinical source feature, and ||·||_2 represents the L2 norm, i.e., the Euclidean distance. A larger semantic distance indicates a greater difference in feature representation between the two modalities, while a smaller distance indicates a greater similarity. The semantic distance is decomposed dimension by dimension, calculating the distance contribution on each feature dimension, i.e., calculating the absolute value of the difference between the image source feature and the clinical source feature on each dimension. The distance contributions of the D dimensions are ranked, and the dimension with the largest contribution is identified. A complementarity gap threshold is set as the 75th quantile of the distance contributions of all dimensions; dimensions with a distance contribution greater than the threshold are marked as modal complementarity gap dimensions. The modal complementarity gap dimension corresponds to two feature dimensions that differ significantly between modalities, representing information that one modality possesses while the other lacks. For the diagnosis of orthopedic infections, the imaging modality has an advantage in the dimensions of three-dimensional morphology of bone destruction and periosteal reaction type, while the clinical modality lacks this information. Conversely, the clinical modality has an advantage in the dimensions of dynamic changes in inflammatory markers (such as the trend of CRP values ​​from initial diagnosis to post-treatment), body temperature curves, and symptom duration, while the imaging modality lacks this information. These dimensions constitute the modal complementarity gap dimension.

[0057] Gap-filling analysis was performed along the modal complementarity gap dimension to determine the type of complementary enhancement. The modal complementarity gap dimension includes several feature dimensions, on which the feature values ​​of the two modalities show significant differences. For each complementary gap dimension, the numerical relationship between the image source feature and the clinical source feature was analyzed. If the value of the image source feature is significantly greater than that of the clinical source feature, it is marked as an image-dominant gap; if the value of the clinical source feature is significantly greater than that of the image source feature, it is marked as a clinical-dominant gap. A significance threshold of 2 times was set, meaning that a significant advantage was considered to exist when the value of one modality was more than twice that of another. Based on the degree of advantage, the complementary gap dimension was classified into three types of complementary enhancement: strong complementarity, moderate complementarity, and weak complementarity, corresponding to modal synergy of 0.9, 0.6, and 0.3, respectively. For the orthopedic infection diagnosis task, the strong complementarity dimension may account for 30%, the moderate complementarity dimension for 40%, and the weak complementarity dimension for 30%.

[0058] Complementary enhancement types are quantified into modal synergy and mapped to generate differential adaptive weights. For each feature dimension, a modal synergy value is assigned according to its complementary enhancement type. The modal synergy of strong complementarity is 0.9, that of medium complementarity is 0.6, and that of weak complementarity is 0.3. For non-complementary gap dimensions, i.e., dimensions where the feature values ​​of the two modalities are close, the modal synergy is set to 1.0, indicating that the two modalities are completely consistent in that dimension. The modal synergy vector has a dimension of 1×D, and each element reflects the degree of synergy between the two modalities in that feature dimension. Differential adaptive weights are generated based on the modal synergy, and the weight calculation formula is W_adapt(k)=(1+C(k)) / 2, where W_adapt(k) is the differential adaptive weight of the k-th dimension, and C(k) is the modal synergy of the k-th dimension, with values ​​ranging from 0.3 / 0.6 / 0.9 / 1.0, and the weight range is from 0.65 to 1.0. Dimensions with high synergy should be weighted close to 1.0, indicating that the two modalities should be fully integrated. Dimensions with low synergy should be weighted close to 0.65, indicating that the two modalities should be integrated moderately.

[0059] Adaptive weighted fusion based on differential adaptive weights is used to determine the multimodal joint representation. The interaction features are 1×D in dimension, and the differential adaptive weights are also 1×D in dimension. A corresponding differential adaptive weight is applied to each dimension of the interaction features, and the weighted fusion calculation formula is F_joint(k) = W_adapt(k) × F_inter(k), where F_joint(k) is the k-th dimension of the multimodal joint representation, W_adapt(k) is the differential adaptive weight of the k-th dimension, and F_inter(k) is the k-th dimension of the interaction feature. The differential adaptive weights approach 1.0 in dimensions with high synergy, fully preserving the information of the interaction features, while moderately attenuating the contribution of the interaction features in dimensions with low synergy. The multimodal joint representation is 1×D in dimension, integrating information from clinical feature vectors and image feature matrices. It achieves bidirectional semantic interaction through a cross-attention mechanism and differential-perceptive adaptive fusion through modality consistency assessment. For the diagnosis of orthopedic infections, multimodal combined representation integrates the morphological features of bone destruction in the lesion, the imaging manifestations of periosteal reaction and soft tissue inflammation, as well as the patient's inflammatory markers and clinical risk factors, and has stronger discriminative ability compared with single-modal representation.

[0060] Step S140: Quantize the uncertainty of the multimodal joint representation to generate a confidence distribution, extract high-confidence features from the confidence distribution, construct classification decision rules according to the high-confidence features, and generate preliminary classification results based on the classification decision rules.

[0061] Specifically, uncertainty quantification is performed on the multimodal joint representation to generate a confidence distribution. The multimodal joint representation contains feature information fused from clinical and imaging modalities, with a dimension of 1×D. Bayesian deep learning is used to quantify the uncertainty of the multimodal joint representation, thus quantifying the degree of uncertainty in the model's predictions. The multimodal joint representation is input into a classifier, and multiple forward propagations are performed by introducing randomness to generate multiple prediction results. Statistical analysis is performed on the multiple prediction results to calculate the prediction probability distribution for each category. The prediction probability distribution reflects the uncertainty of the model's judgment on different categories; a more concentrated probability distribution indicates a more certain prediction, while a more dispersed probability distribution indicates a more uncertain prediction. For each feature dimension of the multimodal joint representation, the confidence score of that dimension's contribution to the classification result is calculated using the formula C(k) = μ_k / (μ_k + σ_k), where C(k) is the confidence score of the k-th feature dimension, μ_k is the mean contribution of that dimension in multiple predictions, and σ_k is the standard deviation of the contribution. The confidence score ranges from 0 to 1. A higher confidence score indicates a more stable and reliable contribution of that feature dimension to the classification result, while a lower confidence score indicates a larger fluctuation in the contribution of that feature dimension across different predictions. The confidence scores are calculated for each of the D feature dimensions to obtain the confidence score distribution. For orthopedic infection diagnosis tasks, feature dimensions closely related to the degree of bone destruction and the type of periosteal reaction typically have higher confidence scores, while feature dimensions related to background bone tissue have lower confidence scores.

[0062] In some embodiments, extracting high-confidence features from the confidence distribution includes: receiving confidence values ​​of all features in the confidence distribution; setting a screening threshold based on the confidence values ​​to define high-confidence intervals and low-confidence intervals; extracting corresponding features within the high-confidence intervals, performing a perturbation resistance test to generate a candidate feature set; and determining high-confidence features from the candidate feature set in combination with potential diagnostic clues from the low-confidence intervals.

[0063] Receive confidence scores for all features in the confidence distribution. Extract confidence scores for all D feature dimensions from the confidence distribution. Perform a range check on the confidence scores to confirm that all scores are within the range of 0 to 1. Calculate the mean and standard deviation of the confidence scores. The mean confidence score reflects the average reliability level of the features, and the standard deviation reflects the dispersion of the confidence scores. For orthopedic infection diagnosis tasks, confidence scores typically exhibit a bimodal distribution. Some features have high confidence scores corresponding to discriminative features closely related to the infected lesions (such as the degree of bone destruction, periosteal reaction type, and CRP level), while other features have low confidence scores corresponding to redundant features related to normal bone tissue or background noise. Record the statistical characteristics of the confidence distribution, including the mean, standard deviation, minimum, and maximum values, to facilitate subsequent threshold setting and extraction of high-confidence features.

[0064] A screening threshold was set based on confidence levels to define high-confidence and low-confidence intervals. A statistical method was used to set the screening threshold, calculated as T_conf = μ_conf + 0.5 × σ_conf, where T_conf is the confidence screening threshold, μ_conf is the mean of the confidence distribution, σ_conf is the standard deviation of the confidence distribution, and a coefficient of 0.5 is used to adjust the leniency of the threshold. Feature dimensions with confidence levels greater than the screening threshold were assigned to the high-confidence interval, while those with confidence levels less than or equal to the screening threshold were assigned to the low-confidence interval. High-confidence intervals correspond to feature dimensions whose model predictions are stable and reliable, and these dimensions make a clear and consistent contribution to the classification results. Low-confidence intervals correspond to feature dimensions whose model predictions have greater uncertainty, and the contributions of these dimensions fluctuate significantly across different predictions. The feature dimension index sets for high-confidence and low-confidence intervals were recorded. The semantic distribution of features in the high-confidence and low-confidence intervals was analyzed. Features in high-confidence intervals typically correspond to the degree of bone destruction, type of periosteal reaction, extent of soft tissue swelling in imaging, and highly discriminative indicators such as CRP levels and white blood cell counts in clinical practice. Features in low-confidence intervals typically correspond to normal bone tissue, noise components, or redundant information that has little impact on classification results.

[0065] For example, the step of extracting corresponding features within the high confidence interval and performing a perturbation resistance test to generate a candidate feature set includes: performing a perturbation resistance test on the corresponding features extracted within the high confidence interval to generate a perturbation resistance score; identifying perturbation-sensitive dimensions and screening high-stability features based on the perturbation resistance score; extracting key discriminant features of the perturbation-sensitive dimensions and combining them with the high-stability features to generate a feature importance map; and determining a candidate feature set based on the feature importance map.

[0066] Perturbation resistance testing is performed on the corresponding features extracted within the high confidence interval to generate a robustness score. Perturbation testing is conducted individually for each feature dimension to evaluate the stability of the feature under input noise or variation. For the k-th feature dimension, random noise of varying intensities is added to the original feature values ​​to construct a perturbation sample set. The original multimodal joint representation and the perturbed representation are input into the classifier to obtain multiple prediction results. Consistency analysis is performed on the prediction results to calculate the consistency rate of the predicted categories and the variance of the predicted probabilities. The robustness score is calculated using the formula R(k) = w_c × consistency(k) + w_p × stability(k), where R(k) is the robustness score of the k-th feature dimension, consistency(k) is the consistency rate of the predicted categories, stability(k) is the stability of the predicted probabilities, and w_c and w_p are the weighting coefficients for consistency and stability, respectively. The consistency rate reflects whether the predicted categories remain consistent before and after perturbation, while the stability reflects the degree of fluctuation in the predicted probabilities. The immunity score ranges from 0 to 1, with a higher score indicating greater feature stability and a lower score indicating greater sensitivity to disturbances. For orthopedic infection diagnosis, features related to bone destruction boundaries and periosteal reaction continuity typically have high immunity scores because these are infection-specific manifestations and less susceptible to noise interference. In contrast, features related to image contrast, window width, and window level have lower immunity scores and are more easily affected by changes in imaging parameters.

[0067] Based on the robustness score, perturbation-sensitive dimensions are identified and high-stability features are selected. A threshold screening method is applied to the robustness score, setting it to 0.8. Feature dimensions with scores greater than 0.8 are marked as high-stability features, while those with scores less than or equal to 0.8 are marked as perturbation-sensitive dimensions. Perturbation-sensitive dimensions correspond to features that strongly react to input perturbations. While these features have high confidence, their stability is insufficient, potentially leading to fluctuations in classification results under different input conditions. High-stability features correspond to features that possess both high confidence and strong robustness; these are the most reliable classification criteria. The number and distribution patterns of perturbation-sensitive dimensions are statistically analyzed, and their semantic characteristics are examined. Perturbation-sensitive dimensions typically correspond to subtle texture changes, edge transition regions, or contrast-sensitive areas in images, which are easily affected by imaging parameters and noise. High-stability features are selected, and their dimensional index sets are recorded. Further analysis of the high-stability features is performed, calculating the contribution weight of each feature dimension to the classification results. For the diagnosis of orthopedic infections, high stability features are mainly concentrated on the macroscopic morphological characteristics of the lesion, such as the extent of bone destruction, the type of periosteal reaction, and the degree of soft tissue swelling, as well as clinical hard indicators such as CRP level, white blood cell count, and body temperature, which are not easily affected by measurement errors.

[0068] Key discriminative features of perturbation-sensitive dimensions are extracted and combined with highly stable features to generate a feature importance map. Although the overall stability of perturbation-sensitive dimensions is insufficient, they may contain discriminative features with key value for specific diagnostic scenarios. The discriminative ability of features in perturbation-sensitive dimensions is analyzed using a feature importance evaluation method. The information gain of each perturbation-sensitive dimension to the classification result is calculated; the information gain reflects the contribution of the feature to reducing classification uncertainty. Perturbation-sensitive dimensions with large information gains contain key discriminative information; although their stability is insufficient, their discriminative value is high, and they are extracted as key discriminative features. Highly stable features and key discriminative features are integrated to construct a feature importance map. The feature importance map assigns an importance score to each feature dimension, and the score calculation formula is I(k) = w_stable(k) + β × w_key(k), where I(k) is the importance score of the k-th feature dimension, w_stable(k) is the weight of the highly stable feature, w_key(k) is the weight of the key discriminative feature normalized according to the information gain, and β is the key feature weight coefficient set to 0.5. The feature importance map is visualized, and the importance distribution of different feature dimensions is displayed in the form of a heatmap, which facilitates the analysis and interpretation of the basis for classification decisions.

[0069] Candidate feature sets are determined based on feature importance maps. These maps contain importance scores for all feature dimensions, which combine feature stability and discriminative power. Thresholding is applied to the feature importance maps, and feature dimensions with higher importance scores are included in the candidate feature set. The candidate feature set combines high confidence, high stability, and key discriminative power; these features contribute most to the classification task and provide stable and reliable predictions. Redundancy analysis is performed on the candidate feature set, calculating the correlation between candidate features and removing highly correlated redundant features to reduce feature dimensions while maintaining discriminative power. The composition structure of the candidate feature set is analyzed, and the proportion of features from imaging modalities and clinical modalities is statistically analyzed to evaluate the relative contributions of the two modalities to the classification task. Imaging modal features mainly correspond to imaging manifestations such as bone destruction morphology, periosteal reaction type, and soft tissue swelling, while clinical modal features mainly correspond to clinical risk factors such as patient body temperature, inflammatory markers, and symptom duration. For orthopedic infection diagnosis tasks, imaging features typically account for 60-70% and clinical features for 30-40% of the candidate feature set, reflecting the dominant role of imaging morphology features in the diagnosis of orthopedic infections.

[0070] High-confidence features are determined by combining the candidate feature set with potential diagnostic clues from low-confidence intervals. The candidate feature set contains high-quality feature dimensions that have undergone multiple screenings, while the low-confidence intervals contain feature dimensions with lower confidence. Although the features in the low-confidence intervals have greater overall uncertainty, they may contain potential clues with diagnostic value for specific cases. For example, certain rare imaging signs or abnormal clinical indicators, although having low confidence in the overall dataset, may be important discriminative markers in specific cases. The association between features in the low-confidence intervals and the candidate feature set is analyzed to identify low-confidence features containing potential diagnostic clues. If a low-confidence feature is significantly associated with multiple high-confidence features in the candidate feature set, it indicates that the feature may carry supplementary diagnostic information and can provide additional discriminative clues when combined with high-confidence features. The final high-confidence features are determined based on a comprehensive analysis of the candidate feature set and potential diagnostic clues. High-confidence features comprehensively consider the feature's confidence, stability, discriminative power, and synergistic effect with other features. They include both core features that are stable and reliable in most cases and appropriate supplementary features that are of key value to specific case types, making them the most reliable and comprehensive classification basis.

[0071] Classification decision rules are constructed based on high-confidence features. A decision tree algorithm is used, with high-confidence features as input and disease category as output, to train the decision tree model. Internal nodes of the decision tree correspond to the judgment conditions of the feature dimensions, and leaf nodes correspond to the classification results. For a binary classification task of orthopedic infection diagnosis, the root node of the decision tree might branch first based on CRP level features, with patients having CRP greater than 100 mg / L tending to have acute infection. The second-level nodes might be based on bone destruction features, with cases showing clear bone destruction having a high probability of infection. The third-level nodes might be based on body temperature and white blood cell count. The decision tree is constructed layer by layer until a preset tree depth or leaf node purity requirement is met. The trained decision tree is pruned, removing branches that contribute less to classification and simplifying the decision rules. The decision tree is then converted into a rule set, with each path from the root node to a leaf node corresponding to a classification rule. The classification rules take the form of IF condition THEN category, for example, IF CRP>100mg / L AND bone destruction AND body temperature>38.5℃ AND white blood cell count>12000 / μL THEN acute orthopedic infection. The condition part consists of judgment conditions based on high-confidence features. The rule set is optimized by merging similar rules and removing redundant rules to ensure its conciseness and efficiency. The construction of classification decision rules transforms the classification process from a black-box model into an interpretable rule system, improving the reliability of diagnostic results.

[0072] Preliminary classification results are generated based on classification decision rules. The patient's multimodal joint representation is input into the classification decision rule system, and feature values ​​corresponding to the high-confidence feature set are extracted to form a high-confidence feature vector. The high-confidence feature vector is matched with the classification decision rules, and the condition parts of each rule are checked one by one. For each rule, it is determined whether the high-confidence feature vector satisfies all the conditions in the rule. If all the conditions of a rule are satisfied, the rule is activated, and the category corresponding to the rule is output as the prediction result. If multiple rules are activated but the predicted categories are inconsistent, the category supported by the most rules is selected as the final prediction. The confidence of the preliminary classification result is calculated using the formula Conf_pred = N_support / N_activated, where Conf_pred is the confidence of the prediction result, N_support is the number of rules supporting the predicted category, and N_activated is the total number of activated rules. The confidence range is from 0 to 1; the higher the confidence, the more reliable the prediction result is as multiple rules consistently support it. Prediction results with low confidence are marked as indeterminate cases, and further investigation is recommended. The system records the list of activated rules and the contribution of each rule to the prediction, providing doctors with a basis for decision-making. The preliminary classification results include three parts: prediction category, prediction confidence, and the list of activated rules.

[0073] Step S150: Identify difficult samples based on the preliminary classification results, extract boundary feature patterns from the difficult samples to generate risk stratification criteria, construct a hierarchical diagnosis system based on the risk stratification criteria, and output hierarchical diagnosis results based on the hierarchical diagnosis system.

[0074] Specifically, difficult samples are identified based on the preliminary classification results. The preliminary classification results include predicted categories, prediction confidence, and a list of activation rules. The prediction confidence and rule consistency of each sample in the preliminary classification results are analyzed, and samples with high classification uncertainty are identified as difficult samples. The criteria for determining difficult samples include cases with low prediction confidence, multiple activated rules but inconsistent predicted categories, or ambiguous judgments of key features. Prediction uncertainty is measured by the distribution characteristics of the prediction probabilities; a more concentrated probability distribution indicates a more certain prediction, while a more dispersed probability distribution or one close to the decision boundary indicates higher uncertainty. Difficult samples correspond to cases with high prediction uncertainty by the classifier. These cases are usually located near the category decision boundary or have atypical clinical and imaging manifestations, making it difficult for the automatic classification system to make a clear judgment. For orthopedic infection diagnosis tasks, difficult samples may include borderline cases such as bone changes with blurred imaging boundaries, chronic infections with osteosclerosis, and mildly abnormal inflammatory markers but atypical imaging signs. The identification of difficult samples provides a key focus for subsequent boundary feature analysis, risk stratification, and graded diagnosis.

[0075] In some embodiments, the step of extracting boundary feature patterns from the difficult samples to generate risk stratification criteria includes: projecting the difficult samples onto the decision space to locate the boundary neighborhood; detecting local gradient changes in the boundary neighborhood to identify boundary-sensitive feature dimensions; extracting key decision components along the boundary-sensitive feature dimensions to construct boundary feature patterns; and quantifying boundary sensitivity based on the boundary feature patterns to classify risk levels and form risk stratification criteria.

[0076] Difficult samples are projected onto the decision space to locate their boundary neighborhood. The decision space is a low-dimensional space composed of the classifier's output probabilities. For binary classification tasks, the decision space is two-dimensional, with the horizontal axis representing the non-infection probability and the vertical axis representing the infection probability. The multimodal joint representation of each difficult sample is input into the classifier to obtain the sample's class prediction probability vector. For binary classification of orthopedic infection diagnosis, the prediction probability vector contains two elements: the infection probability and the non-infection probability. The prediction probability vector is used as coordinates to mark the position of the difficult sample in the decision space. The decision boundary corresponds to the line where the infection probability equals the non-infection probability, i.e., the position with a probability of 0.5. The distance from each difficult sample to the decision boundary is calculated using the formula d(i) = |P_pos(i) - P_neg(i)|, where d(i) is the distance from the i-th difficult sample to the decision boundary, P_pos(i) is the positive class probability, and P_neg(i) is the negative class probability. A smaller distance indicates that the sample is closer to the decision boundary, and the greater the classification uncertainty. A boundary neighborhood threshold of 0.2 is set, and difficult samples with a distance less than 0.2 are included in the boundary neighborhood. The boundary neighborhood contains difficult samples that are closest to the decision boundary, and the classification of these samples is the most ambiguous. For the diagnosis of orthopedic infections, the samples in the boundary neighborhood may include cases with an infection probability of 0.45 and a non-infection probability of 0.55. The close probabilities of the two classes lead to diagnostic difficulties.

[0077] This study identifies boundary-sensitive feature dimensions by detecting local gradient changes in the boundary neighborhood. The boundary neighborhood contains the hard samples closest to the decision boundary. For each sample in the boundary neighborhood, the gradient of its multimodal joint representation with respect to the classification result is calculated. The gradient reflects the degree to which small changes in the feature dimension affect the predicted probability. The backpropagation algorithm is used to calculate the gradient vector of the classifier output relative to the multimodal joint representation. Each element of the gradient vector corresponds to a feature dimension; the larger the absolute value of the element, the more sensitive that dimension is to the classification result. Statistical analysis is performed on the gradient vectors of all samples in the boundary neighborhood to calculate the average absolute gradient value of each feature dimension. Boundary-sensitive feature dimensions are identified when the average absolute gradient value of the feature dimension is greater than the threshold G_threshold = μ_grad + σ_grad, where G_threshold is the gradient threshold, μ_grad is the mean of the average gradients of all feature dimensions, and σ_grad is the standard deviation. Feature dimensions with gradient absolute values ​​greater than the threshold are marked as boundary-sensitive feature dimensions. Boundary-sensitive feature dimensions are those that significantly affect the classification decision and change drastically near the decision boundary. For the diagnosis of orthopedic infections, boundary-sensitive features may include morphological features such as the clarity of bone destruction boundaries, the continuity of periosteal reaction, and the extent of soft tissue swelling. Small changes in these features may lead to a reversal of the infection diagnosis.

[0078] Boundary feature patterns are constructed by extracting key decision components along the boundary-sensitive feature dimension. The boundary-sensitive feature dimension has a significant impact on classification decisions. Feature values ​​corresponding to the boundary-sensitive feature dimension are extracted from the multimodal joint representation to form a boundary-sensitive feature vector. Principal component analysis (PCA) is performed on the boundary-sensitive feature vector to identify the main directions of change among features. PCA reduces the dimensionality of the high-dimensional boundary-sensitive feature vector to a low-dimensional space, extracting the principal components that best explain the feature variance. A principal component retention threshold is set at a cumulative variance contribution rate of 85%, typically retaining the top 3-5 principal components. Each principal component corresponds to a key decision component, reflecting the change pattern of the boundary-sensitive feature in that direction. The loading vectors of each principal component are analyzed; the elements of the loading vector represent the weights of each boundary-sensitive feature dimension on that principal component. Feature dimensions with an absolute weight greater than 0.3 are marked as the main contributing features of that principal component. A boundary feature pattern is constructed, which includes the direction vectors of the principal components and their corresponding variance contribution rates. The boundary feature pattern describes the feature change patterns of hard samples in the vicinity of the boundary. For the diagnosis of orthopedic infections, the first principal component may correspond to the pattern of changes in bone destruction morphology, the second principal component may correspond to the pattern of changes in periosteal reaction type and soft tissue swelling, and the third principal component may correspond to the pattern of synergistic changes in inflammatory markers and imaging signs.

[0079] Risk levels are categorized based on boundary sensitivity quantification using boundary feature patterns, forming a basis for risk stratification. Boundary feature patterns describe the characteristic variation patterns of difficult samples near the decision boundary. For each difficult sample, the projection coefficients of the sample onto each principal component of the boundary feature pattern are calculated. The projection coefficients reflect the degree of deviation of the sample in the direction of that principal component. Boundary sensitivity is quantified using the formula S_boundary=Σ(w_pc×|proj_pc|), where S_boundary is the boundary sensitivity, w_pc is the variance contribution rate of the pc-th principal component as a weight, proj_pc is the projection coefficient of the sample onto the pc-th principal component, and Σ represents the summation over all retained principal components. Boundary sensitivity reflects the comprehensive degree of deviation of the sample from the decision boundary; higher sensitivity indicates that the sample is close to the boundary in multiple directions of the boundary feature pattern, and the diagnostic risk is greater. Risk levels are categorized based on boundary sensitivity: high risk corresponds to a boundary sensitivity greater than 0.6, medium risk corresponds to a boundary sensitivity between 0.3 and 0.6, and low risk corresponds to a boundary sensitivity less than 0.3. High-risk samples are located closest to the decision boundary, exhibiting the greatest classification uncertainty and requiring special attention or further investigation. Medium-risk samples are relatively close to the decision boundary, presenting some diagnostic uncertainty; it is recommended to combine them with other auxiliary information for a comprehensive judgment. Low-risk samples, although labeled as difficult samples, are relatively far from the decision boundary, making their classification relatively reliable. The number and proportion of samples at each risk level are statistically analyzed to establish a basis for risk stratification. For orthopedic infection diagnosis tasks, high-risk samples may account for 30% of difficult samples, medium-risk samples for 50%, and low-risk samples for 20%.

[0080] A tiered diagnostic system is constructed based on risk stratification. Risk stratification classifies samples into three levels: high-risk, medium-risk, and low-risk. The tiered diagnostic system sets different diagnostic procedures and decision-making criteria for each risk level. For low-risk samples, a standard diagnostic procedure is used, directly providing a diagnostic conclusion based on the preliminary classification results, requiring a confidence level greater than 0.7. The diagnostic confidence of low-risk samples is relatively high, usually requiring no additional diagnostic steps. For medium-risk samples, an enhanced diagnostic procedure is used, introducing additional feature analysis and a multi-expert voting mechanism based on the preliminary classification results, suitable for samples with confidence levels between 0.5 and 0.7. Values ​​on boundary-sensitive feature dimensions are extracted from medium-risk samples, and the relationship between these features and infection diagnosis is analyzed to provide supporting evidence for diagnosis. For high-risk samples, an expert consultation procedure is used; cases marked as difficult to diagnose require human expert intervention, suitable for samples with confidence levels less than 0.5. High-risk samples have the greatest classification uncertainty, and the reliability of automated diagnostic systems is insufficient, requiring experienced clinical experts to make comprehensive judgments based on image interpretation and clinical history. The tiered diagnostic system rationally allocates diagnostic resources to samples of different risk levels, enabling rapid and automated diagnosis for low-risk samples and ensuring in-depth involvement of human experts for high-risk samples, thus balancing diagnostic efficiency and accuracy.

[0081] The system outputs stratified diagnostic results based on a tiered diagnostic system. Following the diagnostic procedures set for different risk levels within the system, the multimodal joint representation of the sample to be diagnosed is input into a classifier to obtain preliminary classification results and prediction confidence. The risk level is determined based on the sample's boundary sensitivity. Based on the rules of the tiered diagnostic system, corresponding diagnostic procedures are executed for samples at different risk levels. The output format of the stratified diagnostic results is adaptively adjusted according to the risk level, reflecting the differentiated strategy of the tiered diagnostic system: low-risk samples output a concise report including predicted category and confidence, with a simple and intuitive format for rapid clinical decision-making; medium-risk samples output a detailed report including predicted category, confidence, boundary feature analysis, and diagnostic recommendations. The boundary feature analysis explains the relationship between key features and the diagnostic conclusion, and the diagnostic recommendations include specific plans for combining other examinations or follow-up observations; high-risk samples output a complete report including multiple prediction probabilities, probability distributions, uncertainty source analysis, and expert consultation recommendations, marked as difficult cases and listing contradictory or ambiguous features, providing comprehensive decision support information for expert consultation. For diagnosing orthopedic infections, low-risk samples show high accuracy with automated diagnosis and can be directly used to guide clinical decision-making and anti-infection treatment. Medium-risk samples require additional diagnostic steps, such as boundary feature analysis and multi-expert voting, to improve accuracy. High-risk samples require human expert involvement, combining clinical experience, etiological examination, and multimodal imaging information for comprehensive judgment. If necessary, biopsy should be performed for pathological confirmation to ensure diagnostic quality. This stratified diagnostic approach enables refined management of the diagnostic process, providing different levels of diagnostic information based on the degree of sample uncertainty. This fully reflects the stratified strategy of the hierarchical diagnostic system and improves the overall reliability and practicality of the diagnostic system.

[0082] To implement the multimodal orthopedic infection diagnosis method based on dual-stream encoder fusion corresponding to the above method embodiments, and to achieve the corresponding functions and technical effects. See also Figure 2 , Figure 2 This diagram illustrates the structure of a multimodal orthopedic infection diagnostic system 200 based on dual-stream encoder fusion according to an embodiment of this application. For ease of explanation, only the parts relevant to this embodiment are shown. The multimodal orthopedic infection diagnostic system 200 based on dual-stream encoder fusion provided in this embodiment includes:

[0083] The data acquisition module 201 is used to acquire multi-window medical image data and clinical indicator data, generate an image feature matrix from the multi-window medical image data through an image encoder, and generate a clinical feature vector from the clinical indicator data through a clinical encoder.

[0084] Image processing module 202 is used to identify cross-slice context features using the image feature matrix through a global transformer, calculate slice importance weights for the cross-slice context features to generate weighted pooling coefficients, and perform adaptive pooling processing on the cross-slice context features using the weighted pooling coefficients to determine the image embedding representation.

[0085] The fusion processing module 203 is used to perform a fully connected transformation on the clinical feature vector to generate a clinical embedding representation, and to perform bidirectional semantic interaction between the clinical embedding representation and the image embedding representation through a cross-attention mechanism to generate interactive features. The interactive features are then subjected to modality consistency evaluation to quantify modality difference and generate difference adaptive weights. Based on the difference adaptive weights, adaptive weighted fusion is performed to determine the multimodal joint representation.

[0086] The classification decision module 204 is used to perform uncertainty quantization on the multimodal joint representation to generate a confidence distribution, extract high-confidence features from the confidence distribution, construct classification decision rules according to the high-confidence features, and generate preliminary classification results based on the classification decision rules;

[0087] The risk stratification module 205 is used to identify difficult samples based on the preliminary classification results, extract boundary feature patterns from the difficult samples to generate risk stratification criteria, construct a hierarchical diagnosis system based on the risk stratification criteria, and output stratified diagnosis results based on the hierarchical diagnosis system.

[0088] The aforementioned multimodal orthopedic infection diagnostic system 200 based on dual-stream encoder fusion can implement the multimodal orthopedic infection diagnostic method based on dual-stream encoder fusion described in the above method embodiments. The options in the above method embodiments are also applicable to this embodiment and will not be detailed here. The remaining content of this application's embodiments can be referred to the content of the above method embodiments, and will not be repeated in this embodiment.

[0089] The purpose of the above embodiments is to reproduce and derive the technical solution of the present invention by way of example, and to fully describe the technical solution, purpose and effect of the present invention. The purpose is to enable the public to have a more thorough and comprehensive understanding of the disclosure of the present invention, and not to limit the scope of protection of the present invention.

[0090] The above embodiments are not an exhaustive list based on the present invention, and there may be many other embodiments not listed. Any substitutions and improvements made without departing from the concept of the present invention are within the protection scope of the present invention.

Claims

1. A multimodal orthopedic infection diagnosis method based on dual-stream encoder fusion, characterized in that, include: Collect multi-window medical image data and clinical indicator data, generate an image feature matrix from the multi-window medical image data using an image encoder, and generate a clinical feature vector from the clinical indicator data using a clinical encoder. The image feature matrix is ​​used to identify cross-slice context features through a global transformer. Slice importance weights are calculated for the cross-slice context features to generate weighted pooling coefficients. Adaptive pooling processing is performed on the cross-slice context features using the weighted pooling coefficients to determine the image embedding representation. The clinical feature vector is transformed using a fully connected method to generate a clinical embedding representation. A cross-attention mechanism is then used to enable bidirectional semantic interaction between the clinical embedding representation and the image embedding representation to generate interactive features. This process includes: retrieving the image embedding representation using the clinical embedding representation to generate a clinical-guided attention map; retrieving the clinical embedding representation using the image embedding representation to generate an image-guided attention map; comparing the clinical-guided attention map and the image-guided attention map to identify modal asymmetry dominant modes and generate dominant transition features; adjusting diagnostic preferences based on the dominant transition features to generate interactive features; performing modal consistency assessment on the interactive features to quantify modal differences and generate adaptive difference weights; and performing adaptive weighted fusion based on the adaptive difference weights to determine a multimodal joint representation. Specifically, the step of comparing the clinical-guided attention map and the image-guided attention map to identify modal asymmetry dominant modes and generate dominant transition features includes: calculating the response intensity difference between the clinical-guided attention map and the image-guided attention map to generate an asymmetric response distribution; identifying the dominant modality region from the asymmetric response distribution; performing influence propagation analysis on the dominant modality region to construct a modal dominance topology; and extracting cross-modal transmission paths based on the modal dominance topology to generate dominant transition features. Uncertainty quantization is performed on the multimodal joint representation to generate a confidence distribution. High-confidence features are extracted from the confidence distribution. Classification decision rules are constructed according to the high-confidence features. Preliminary classification results are generated based on the classification decision rules. Based on the preliminary classification results, difficult samples are identified, boundary feature patterns are extracted from the difficult samples to generate risk stratification criteria, a hierarchical diagnosis system is constructed based on the risk stratification criteria, and hierarchical diagnosis results are output based on the hierarchical diagnosis system.

2. The method according to claim 1, characterized in that, The step of generating an image feature matrix from the multi-window medical image data via an image encoder includes: The multi-window medical image data is split into image sequences for each window. Local spatial features are extracted by applying a shared-weight convolutional network slice by slice to each window image sequence; Perform inter-window comparison tests on the local spatial features to identify areas of concentrated differences and mark them as key areas of interest; The local spatial features and the key locations of interest are combined to form an image feature matrix.

3. The method according to claim 1, characterized in that, The step of calculating slice importance weights and generating weighted pooling coefficients for the cross-slice context features includes: Measure the information entropy value of each slice for the cross-slice context features; The positions of high-information-content slices and low-information-content slices are identified by sorting according to the information entropy values. An attention weight mapping is established by calculating the slice importance weights based on the high-information-content slice positions; The attention weight mapping is used in conjunction with the low-information slice position to generate weighted pooling coefficients.

4. The method according to claim 1, characterized in that, The step of evaluating modal consistency of the interaction features, quantifying modal dissimilarity, and generating adaptive dissimilarity weights includes: Based on the interaction features, image source features and clinical source features are separated; Semantic distance measurement is performed on the image source features and the clinical source features to generate modality complementarity gap dimensions; The type of complementary enhancement is determined by performing gap-filling analysis along the modal complementary gap dimension. The complementary enhancement type is quantified into modal synergy and mapped to generate differential adaptive weights.

5. The method according to claim 1, characterized in that, Extracting high-confidence features from the confidence distribution includes: Receive the confidence values ​​of all features in the confidence distribution; Based on the aforementioned confidence level values, a filtering threshold is set to define the high confidence interval and the low confidence interval; Within the high confidence interval, corresponding features are extracted, and a disturbance resistance test is performed to generate a candidate feature set; High-confidence features are determined from the candidate feature set by combining potential diagnostic clues from the low-confidence interval.

6. The method according to claim 1, characterized in that, The step of extracting boundary feature patterns from the difficult samples to generate risk stratification criteria includes: The difficult samples are projected into the decision space to locate the boundary neighborhood; Detecting local gradient changes in the boundary neighborhood to identify boundary-sensitive feature dimensions; Extract key decision components along the boundary-sensitive feature dimension to construct a boundary feature pattern; Based on the boundary feature patterns, the boundary sensitivity is quantified to classify risk levels and form a basis for risk stratification.

7. The method according to claim 5, characterized in that, The step of extracting corresponding features within the high confidence interval and performing perturbation resistance testing to generate a candidate feature set includes: Perturbation resistance tests are performed on the corresponding features extracted within the high confidence interval to generate a perturbation resistance score; Based on the disturbance resistance score, disturbance-sensitive dimensions are identified and high-stability features are selected. Key discriminative features of the perturbation-sensitive dimension are extracted and combined with the high-stability features to generate a feature importance map; The candidate feature set is determined based on the feature importance map.

8. A multimodal orthopedic infection diagnostic system based on dual-stream encoder fusion, characterized in that, include: The data acquisition module is used to acquire multi-window medical image data and clinical indicator data, generate an image feature matrix from the multi-window medical image data through an image encoder, and generate a clinical feature vector from the clinical indicator data through a clinical encoder. The image processing module is used to identify cross-slice context features using the image feature matrix through a global transformer, calculate slice importance weights for the cross-slice context features to generate weighted pooling coefficients, and perform adaptive pooling processing on the cross-slice context features using the weighted pooling coefficients to determine the image embedding representation. A fusion processing module is used to perform a fully connected transformation on the clinical feature vector to generate a clinical embedding representation, and to enable bidirectional semantic interaction between the clinical embedding representation and the image embedding representation through a cross-attention mechanism to generate interactive features. This includes: retrieving the image embedding representation using the clinical embedding representation to generate a clinically guided attention map; retrieving the clinical embedding representation using the image embedding representation to generate an image-guided attention map; comparing the clinically guided attention map and the image-guided attention map to identify modal asymmetry dominant modes and generate dominant transition features; adjusting diagnostic preferences based on the dominant transition features to generate interactive features; performing modal consistency assessment on the interactive features to quantify modal differences and generate adaptive difference weights; and performing adaptive weighted fusion based on the adaptive difference weights to determine a multimodal joint representation. Specifically, the step of comparing the clinically guided attention map and the image-guided attention map to identify modal asymmetry dominant modes and generate dominant transition features includes: calculating the difference in response intensity between the clinically guided attention map and the image-guided attention map to generate an asymmetric response distribution; identifying the dominant modality region from the asymmetric response distribution; performing influence propagation analysis on the dominant modality region to construct a modal dominance topology; and extracting cross-modal transmission paths based on the modal dominance topology to generate dominant transition features. The classification decision module is used to perform uncertainty quantification on the multimodal joint representation to generate a confidence distribution, extract high-confidence features from the confidence distribution, construct classification decision rules according to the high-confidence features, and generate preliminary classification results based on the classification decision rules; The risk stratification module is used to identify difficult samples based on the preliminary classification results, extract boundary feature patterns from the difficult samples to generate risk stratification criteria, construct a hierarchical diagnosis system based on the risk stratification criteria, and output hierarchical diagnosis results based on the hierarchical diagnosis system.

Images