A scoliosis explainability screening method based on a multi-region perception prototype network
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- BEIHANG UNIV
- Filing Date
- 2026-04-10
- Publication Date
- 2026-06-23
Smart Images
Figure CN122265728A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of intelligent medical image analysis, specifically relating to an interpretable screening method for scoliosis based on a multi-regional perception prototype network. Background Technology
[0002] Adolescent idiopathic scoliosis (AIS) is a common spinal deformity with a global prevalence of approximately 0.93% to 12%. This condition can cause back pain, impaired cardiopulmonary function, and psychological problems, and its effective clinical management heavily relies on early detection and accurate risk stratification.
[0003] Currently, the gold standard for diagnosing AIS is based on measuring the Cobb angle using standing full-spine X-rays. However, the cumulative radiation from long-term, repeated X-ray monitoring has been proven to be associated with an increased risk of malignant tumors. To mitigate radiation risks, existing technologies have proposed radiation-free methods such as three-dimensional ultrasound imaging and surface topography. However, these methods have high requirements for equipment and operating environments, limiting their application in large-scale population screening.
[0004] Deep learning-based screening methods based on two-dimensional back photographs have become a research hotspot due to their low cost and portability. However, the deep learning models constructed by existing methods mainly adopt a black-box architecture, and the decision-making process lacks transparency. Clinicians cannot confirm whether the model's judgment is based on relevant anatomical features or image confounding factors.
[0005] To address the black-box problem, existing methods typically employ ex post facto interpretations such as saliency maps. However, their reliability in lesion localization in medical imaging is limited, as they are merely approximations of model decisions rather than faithful reflections of the reasoning process.
[0006] Prototype networks, as an inherently interpretable deep learning method, provide a transparent decision-making process through case-based reasoning. However, existing prototype network methods fail to utilize the spatial specificity of scoliosis morphological features, lack position awareness, and cannot effectively capture morphological features of different areas of the back for interpretable screening. Summary of the Invention
[0007] To address the black-box decision-making problem in existing surface screening methods and the lack of location awareness in existing prototype networks, this invention provides an interpretable scoliosis screening method based on a multi-regional sensing prototype network. By using a spinal coordinate encoding mechanism to endow the prototype with spatial location awareness, and by using spinal principal axis spatial constraints to guide the prototype activation to focus on clinically relevant areas, this method provides an interpretable reasoning process based on prototype matching while maintaining classification accuracy.
[0008] The scoliosis interpretability screening method based on a multi-regional sensing prototype network includes the following steps:
[0009] S1. Acquire standardized back surface images I of the subject, adjust the resolution, divide the data into non-overlapping patches, and extract the visual features of each patch to form a feature matrix. .
[0010] Feature matrix ;in For the number of patches, For feature dimensions;
[0011] S2, for the visual feature matrix Perform PCA foreground segmentation to obtain a set of foreground patch indices. and background patch index set ;
[0012] Specifically:
[0013] First, the visual feature matrix Perform principal component analysis on the whole, extract the first principal component, calculate the projection value of the visual features of each patch onto the first principal component and normalize it to the nearest integer. The interval is used to obtain the first principal component projection score for each patch. .
[0014] Then, based on the first principal component projection score of each patch, the foreground determination direction of image I is determined;
[0015] Next, set a threshold for the projection score. A binary foreground mask is generated by combining the determined foreground direction. ;
[0016] If the foreground direction corresponds to a lower The value will satisfy < patch corresponding mask Assign a value of 1 if the foreground direction corresponds to a higher value, otherwise assign a value of 0; The value will satisfy patch corresponding mask Assign a value of 1 if the value is 1, otherwise assign a value of 0.
[0017] Finally, based on the mask The value is used to obtain the foreground patch index set. and background patch index set ;
[0018] S3, Construct the spine reference axis and index the foreground patch. Normalized spine coordinates are calculated for each patch and encoded. Their respective visual features are then fused to obtain the position-aware features of each foreground patch.
[0019] The specific steps are as follows:
[0020] S3.1, Extracting the set The center spatial coordinate set of all corresponding foreground patches is obtained, and principal component analysis is performed on the coordinate set. The direction of the first principal component is taken as the initial estimate of the spine principal axis.
[0021] S3.2, group the central spatial coordinate set by row, calculate the average value of the horizontal coordinates of all foreground patches in each row, and form a coordinate pair with the vertical coordinates of each row to form the spine midline point set;
[0022] S3.3, using the initial estimated parameters of the spinal principal axis as initial values, fits the straight line containing the spinal reference principal axis by minimizing the sum of squared residuals from the set of spinal midline points to the straight line. ;
[0023] These are the line parameters optimized using an iterative optimization algorithm.
[0024] S3.4, calculate the normalized spine coordinates for each foreground patch separately. :
[0025]
[0026] in, This represents the original projection distance of the patch along the longitudinal direction of the spine. This represents the original projection distance of the patch along the transverse direction of the spine.
[0027] S3.5, the normalized spinal coordinates of each patch are mapped to the same coordinates as the visual features using a multilayer perceptron. In a 3D space, each coordinate code is generated and fused into the original patch features using residual connections to obtain position-aware features;
[0028] S4 sets different categories for scoliosis interpretability screening and learns interpretable prototype vectors for each category. Based on the position-aware features of each foreground patch, it calculates the cosine similarity between the patch and each prototype, extracts the spatial maximum activation intensity to obtain the score for each category, and finally outputs the predicted category of image I.
[0029] Specifically:
[0030] First, let the set of categories be... These correspond to four levels: normal, mild, moderate, and severe.
[0031] Then, for each category study An interpretable prototype vector ,common Each prototype is uniquely numbered as follows: ;
[0032] Next, the position-aware features of the nth foreground patch are analyzed. The cosine similarity with each prototype is calculated as follows:
[0033]
[0034] All foreground patches and prototypes cosine similarity set The activation sequence that constitutes this prototype;
[0035] Extract the maximum value from the activation sequence as the activation strength of each prototype: ;
[0036] Based on the activation intensity of each prototype, calculate the maximum activation value of the prototype belonging to each category: ;
[0037] Finally, the predicted category of the current input image is determined based on the maximum activation value of each category:
[0038] .
[0039] For category The corresponding set of prototype indices;
[0040] S5 uses the predicted categories output by the model and the activation sequences of each prototype to generate visual and interpretable screening results;
[0041] Interpretable screening results include the following three parts: output severity grading results, generation of interpretable activation heatmaps, and retrieval of the most similar prototype samples.
[0042] S6. Design a multi-objective joint loss function and use a two-stage training strategy to train the visual Transformer backbone network and update the prototype and coordinate encoder.
[0043] The two-stage training strategy is as follows: in the warm-up stage, the backbone network parameters are frozen, and only the prototype layer and coordinate encoder are trained; in the fine-tuning stage, the backbone network is unfrozen and end-to-end optimization is performed.
[0044] The loss function is as follows:
[0045]
[0046] in, Cross-entropy is used for classification loss; Prototype learning loss; This represents the spatial constraint loss of the spinal principal axis. represents the weighting coefficient for spatial constraint loss.
[0047] The advantages of this invention are:
[0048] 1. Compared with traditional black-box deep learning methods, this invention provides a transparent decision-making process through a prototype matching mechanism, which can show which anatomical regions in the input image have high similarity to typical patterns of the corresponding categories, conforming to clinical reasoning logic and facilitating understanding and verification by clinicians.
[0049] 2. A spinal coordinate encoding mechanism is introduced to transform feature locations into standardized coordinates relative to the spinal axis, enabling the prototype to learn position-sensitive morphological features and effectively capture deformity patterns in different anatomical regions such as the shoulder, scapula, and lumbar region.
[0050] 3. The spinal axis spatial constraint loss is designed to guide the prototype activation to focus on clinically relevant anatomical regions rather than the background, avoiding the model from taking shortcuts through background features and improving interpretability quality.
[0051] 4. The PCA online foreground segmentation module is used, which eliminates the need for additional segmentation annotations or pre-trained segmentation models, reducing data preparation costs and computational complexity.
[0052] 5. It enables radiation-free scoliosis screening, avoiding the cumulative radiation exposure risk of traditional X-ray examinations, and is suitable for long-term monitoring of adolescents and large-scale group screening.
[0053] 6. It achieved excellent performance in both classification accuracy and interpretability metrics, and achieved a good balance between prototype diversity and comprehensiveness in terms of interpretability metrics. Attached Figure Description
[0054] Figure 1 This is a flowchart of an interpretable screening method for scoliosis based on a multi-regional sensing prototype network according to the present invention.
[0055] Figure 2 This is a schematic diagram of the prototype matching classification method used in the present invention. Detailed Implementation
[0056] To facilitate understanding and implementation of the present invention by those skilled in the art, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Obviously, the described embodiments are merely some, not all, embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort should fall within the scope of protection of the present invention.
[0057] This invention employs a deep learning-based interpretability screening method for scoliosis, used to perform radiation-free, interpretable severity grading assessment of adolescent idiopathic scoliosis using back surface photographs. Its core steps include: feature extraction, PCA foreground segmentation, spinal coordinate encoding, prototype matching and classification, interpretability result generation, and model training; specifically:
[0058] S1. Acquire photographs of the subject's back and input them into a pre-trained visual Transformer backbone network to extract patch features; S2. Perform online foreground segmentation of the features using principal component analysis to filter background areas; S3. Calculate the spinal principal axis parameters based on the foreground mask, assign normalized coordinates relative to the spinal principal axis to each foreground patch, generate spinal coordinate codes, and fuse them into the original features; S4. Calculate the cosine similarity between the fused features and prototypes of each severity category, obtain the prototype activation intensity through global max pooling, and output the classification results; S5. Utilize the activation sequences of the predicted categories and each prototype to generate a visual activation heatmap superimposed on the original photographs, and retrieve the most similar historical samples from the training set as an intuitive pathological reference; S6. Train the model using a multi-objective joint loss function that includes classification loss, prototype learning loss, and spinal principal axis spatial constraint loss. This invention enables prototype learning to have positional awareness through spinal principal axis spatial encoding, guides prototype activation to concentrate in clinically relevant areas through spinal principal axis spatial constraints, achieves high-accuracy classification of adolescent scoliosis under radiation-free conditions, and provides a transparent inference process based on prototype matching, solving the problem of opaque decision-making in existing deep learning screening methods.
[0059] like Figure 1 As shown, it includes the following steps:
[0060] S1. Acquire standardized back images (I) of the subject, adjust to a uniform resolution, and input them into a pre-trained visual Transformer backbone network. Divide the network into non-overlapping patches, extract the visual features of each patch, and form a feature matrix. .
[0061] The visual Transformer backbone network divides the input image into segments of size [size missing]. The visual feature matrix is extracted from the non-overlapping patches of pixels. ;in For the number of patches, Let be the feature dimension; denoted as ( ) is the first The visual feature vectors of each patch, i.e., the visual feature matrix The OK.
[0062] S2, statistically analyzing the distribution differences between the foreground and background regions of the image, and applying the visual feature matrix. Perform PCA foreground segmentation to obtain a set of foreground patch indices. and background patch index set ;
[0063] Specifically:
[0064] First, the visual feature matrix Perform principal component analysis on the whole, extract the first principal component, calculate the projection value of the visual features of each patch onto the first principal component and normalize it to the nearest integer. The interval is used to obtain the first principal component projection score for each patch. .
[0065] Then, based on the first principal component projection score of each patch, the foreground determination direction of image I is determined:
[0066] Calculate the corresponding patch in each of the four corner regions (assuming it is the background) of image I. Local mean, and the corresponding values for all patches The overall mean is used to determine if a local mean is higher than the overall mean. If so, the foreground direction corresponds to a value below a set threshold. Value; otherwise, the foreground direction corresponds to a value higher than the set threshold. value.
[0067] Next, set a threshold for the projection score. A binary foreground mask is generated by combining the determined foreground direction. ;
[0068] If the foreground direction corresponds to a lower The value will satisfy < patch corresponding mask Assign a value of 1 if the foreground direction corresponds to a higher value, otherwise assign a value of 0; The value will satisfy patch corresponding mask The mask is assigned a value of 1 if it is not assigned a value of 0 otherwise; morphological closing operations are applied to the mask. Fill isolated gaps inside the foreground and remove edge artifacts from the left and rightmost columns.
[0069] Finally, based on the mask The value is used to obtain the foreground patch index set. and background patch index set ;
[0070] ;
[0071] ;
[0072] S3, Construct the spine reference axis and index the foreground patch. Each patch in the algorithm is calculated based on the foreground mask. Normalized spine coordinates are calculated and encoded, and their respective visual features are fused to obtain the position-aware features of each foreground patch.
[0073] The specific steps are as follows:
[0074] S3.1, Extracting the set The center spatial coordinate set of all corresponding foreground patches is obtained, and principal component analysis is performed on the coordinate set. The direction of the first principal component is taken as the initial estimate of the spine principal axis.
[0075] S3.2, group the central spatial coordinate set by row, calculate the average value of the horizontal coordinates of all foreground patches in each row, and combine it with the corresponding vertical coordinates to form coordinate pairs, thus forming the spine midline point set;
[0076] The average of the x-coordinates of all foreground patches in the i-th row. The vertical coordinate is To form coordinate pairs .
[0077] S3.3, using the initial estimated parameters of the spinal principal axis as initial values, fits the straight line containing the spinal reference principal axis by minimizing the sum of squared residuals from the set of spinal midline points to the straight line. ;
[0078] Line parameters optimized using the BFGS algorithm
[0079] S3.4, calculate the normalized spine coordinates for each foreground patch separately. :
[0080]
[0081] in, These are the normalized coordinates along the longitudinal and transverse directions of the spine, respectively.
[0082] The original projection distance of the patch along the longitudinal direction of the spine: ;
[0083] The original projection distance of the patch along the transverse direction of the spine: ;
[0084] The horizontal center point coordinates for each patch; It is a straight line The center point, where , and These represent the minimum and maximum values of the ordinate in each patch of the foreground region, respectively. is the unit vector along the principal axis of the spine; It is a unit vector perpendicular to the principal axis of the spine; and These represent the maximum absolute values of the original projection distances of all patches in the foreground region along the spine in the longitudinal and transverse directions, respectively.
[0085] S3.5, the normalized spine coordinates of each patch are mapped to the same coordinates as the visual features through a multilayer perceptron containing a two-layer network structure. In a 3D space, each coordinate code is generated and fused into the original patch features using residual connections to obtain position-aware features;
[0086] Normalized spine coordinates of the nth patch Generated coordinate codes fused into the original patch features To obtain location-aware features : .
[0087] S4 sets different categories for scoliosis interpretability screening and learns interpretable prototype vectors for each category. Based on the position-aware features of each foreground patch, it calculates the cosine similarity between the patch and each prototype, extracts the spatial maximum activation intensity to obtain the score for each category, and finally outputs the predicted category of image I.
[0088] Specifically:
[0089] First, let the set of categories be... These correspond to four levels: normal, mild, moderate, and severe.
[0090] Then, for each category study An interpretable prototype vector ,common There are 1 prototype. For ease of representation, all prototypes will be uniformly numbered as follows: ;
[0091] Next, the position-aware features of the nth foreground patch are analyzed. The cosine similarity with each prototype is calculated as follows:
[0092]
[0093] in Foreground patch index, This is the prototype index. Global max pooling is performed only on foreground patches, and all foreground patches are indexed by the prototype. cosine similarity set The activation sequence that constitutes this prototype;
[0094] Extract the maximum value from the activation sequence as the activation strength of each prototype: ;
[0095] Based on the activation intensity of each prototype, calculate the maximum activation value of the prototype belonging to each category: ;
[0096] Finally, the predicted category of the current input image is determined based on the maximum activation value of each category:
[0097] .
[0098] For category The corresponding set of prototype indices;
[0099] S5 uses the predicted categories output by the model and the activation sequences of each prototype to generate visual and interpretable screening results;
[0100] Interpretable screening results include the following three parts:
[0101] Output severity rating results According to Determine the final prediction category and output the confidence score for each category. .
[0102] Generate an interpretable activation heatmap and the activation matrix The middle belongs to the prediction category The activation sequence corresponding to the prototype is summed or maximized. After zero-padding at the corresponding positions of the background region patch in the original mesh, the sequence is reshaped. The spatial distribution map is then bilinearly upsampled to the original image resolution to obtain a visual activation heatmap superimposed on the original back image I.
[0103] Retrieve the most similar prototype sample: for the prediction results The prototype with the highest activation intensity The historical samples with the highest activation intensity of the prototype are retrieved from the training set as reference cases to provide intuitive pathological references based on the similarity of local morphological features.
[0104] The three outputs together constitute an interpretable screening result: the grading result is used to clarify the severity, the heat map can locate the anatomical area of interest of the model, and the reference case provides a basis for comparative reasoning based on the example, thus providing clinicians with transparent and interpretable auxiliary screening results.
[0105] S6. Design a multi-objective joint loss function and use a two-stage training strategy to train the visual Transformer backbone network and update the prototype and coordinate encoder.
[0106] First, a training sample set is constructed, containing several subject body surface images and their corresponding real-world screening category labels of varying degrees. In each training iteration, a sample of [number] samples is extracted from the training sample set. The training samples form a batch input to the backbone network for processing;
[0107] The two-stage training strategy is as follows: in the warm-up stage, the backbone network parameters are frozen, and only the prototype layer and coordinate encoder are trained; in the fine-tuning stage, the backbone network is unfrozen and end-to-end optimization is performed.
[0108] The loss function is as follows:
[0109]
[0110] in, Cross-entropy is used for classification loss; The prototype learning loss is used; clustering loss and separation loss, commonly used in the prototype learning field, are employed to constrain prototype quality. This represents the spatial constraint loss of the spinal principal axis. represents the weighting coefficient for spatial constraint loss.
[0111] Prototype learning loss Includes clustering loss Separation loss and activation equilibrium loss Specifically:
[0112]
[0113] , , These are the weighting coefficients corresponding to each type of loss.
[0114] Clustering loss To promote similarity between sample features and similar prototypes, cosine similarity is used, specifically:
[0115] ;
[0116] in, For sample index within a batch, Batch size; For the sample Real category labels The corresponding set of prototype indices For the sample With prototype The maximum cosine similarity.
[0117] Separation loss The inter-class distinguishability is enhanced by penalizing samples with high similarity to the prototype of the error class, specifically:
[0118] ;
[0119] in This is the maximum allowed similarity threshold between a sample and a prototype from a different class.
[0120] Activate Equilibrium Loss To prevent prototype collapse where only a few prototypes dominate the classification, and to ensure that all prototypes of the same type are fully activated to capture diverse morphological patterns within that severity level, specifically:
[0121] ;
[0122] in, and Samples The maximum and minimum activation values among similar prototypes The maximum allowed activation gap threshold.
[0123] Spinal principal axis spatial constraint loss The calculation method is as follows: An adaptive Gaussian space weight is constructed based on the spinal principal axis.
[0124]
[0125] in, Center point for each patch The vertical distance to the main axis of the spine. Estimated by the second eigenvalue of PCA, This is the width scaling factor.
[0126] The loss function consists of two parts: a penalty for foreground region deviation from the principal axis and a penalty for background region activation.
[0127]
[0128] in, and Each is a current sample A set of patch indices for the foreground and background regions;
[0129] For the current sample Medium prototype In position Positive activation value at that location, For the current sample Cosine similarity; For the current sample In position Gaussian space weights at the location. Example
[0130] S1, Feature Extraction Steps
[0131] First, standardized back images of the subjects were acquired: subjects stood in a natural standing posture, researchers took photos, and all images were uniformly adjusted to [standard format]. Resolution was normalized using ImageNet mean and standard deviation.
[0132] Then, DINOv2-ViT-B / 14 was used as the feature extraction backbone network;
[0133] DINOv2 is a self-distilled visual Transformer that is pre-trained on large-scale unlabeled data and has good transferability in medical image analysis.
[0134] Given an input image DINOv2 classifies it as non-overlapping patches of pixels are obtained Each patch token. After processing by the Transformer encoder, patch features are extracted. ,in The feature dimension is used to retain all patch features to support subsequent local prototype matching, thereby achieving spatial interpretability.
[0135] S2, PCA foreground segmentation step
[0136] Back surface images often contain background regions such as walls and the ground, and this irrelevant information can interfere with prototype learning. This paper designs an online foreground segmentation module based on Principal Component Analysis (PCA) that requires no additional segmentation annotations or pre-trained segmentation models.
[0137] The online foreground segmentation module utilizes the statistical distribution differences between foreground and background regions based on DINOv2 features. For patch features... First, perform PCA dimensionality reduction, taking the projected value of the first principal component and normalizing it to... The interval was reshaped into ,in .
[0138] Then, an adaptive direction detection strategy is used to determine the foreground direction:
[0139] Compare the PCA mean of the four corner regions (assuming they represent the background) with the overall mean. If the mean of the four corner regions is lower than the overall mean, the foreground corresponds to a higher PCA value, and vice versa. (Based on a threshold) Generate a binary foreground mask Morphological closing operations are applied to fill isolated gaps inside the foreground, and the leftmost and rightmost columns are removed to filter out arm edge artifacts.
[0140] According to the mask The value is used to obtain the foreground patch index set. and background patch index set ;
[0141] ;
[0142] ;
[0143] A parameter-frozen DINOv2 copy is introduced specifically for PCA computation, with its weights remaining constant throughout training to ensure the stability and reproducibility of foreground segmentation.
[0144] S3, Spine coordinate encoding steps
[0145] Clinical screening for AIS relies on visual assessment of trunk asymmetry. The TRACE assessment system quantifies trunk morphology across multiple anatomical regions, including the shoulders, scapula, thorax, and lumbar region. These asymmetries relative to the midline of the spine are key visual cues for clinicians to assess the severity of AIS.
[0146] A spinal coordinate encoding module is designed to assign normalized position coordinates relative to the spinal principal axis to each patch, enabling the model to have spatial awareness. This module estimates the spinal principal axis using linear fitting and calculates the spinal coordinates in three stages:
[0147] The first stage is to extract the set. The center spatial coordinate set of all corresponding foreground patches is obtained, and principal component analysis is performed on the coordinate set. The direction of the first principal component is taken as the initial estimate of the spine principal axis.
[0148] In the second stage, the central spatial coordinate set is grouped by row, and the average value of the x-coordinates of all foreground patches in each row is calculated. This average value is then combined with the corresponding y-coordinates to form coordinate pairs. This constitutes the set of points along the midline of the torso.
[0149] In the third stage, using the initial estimated parameters of the spinal principal axis as initial values, the BFGS algorithm is used to optimize the linear parameters. By minimizing the sum of squared residuals from the set of points along the midline of the torso to the straight line, the straight line containing the reference principal axis of the spine is fitted. .
[0150] Based on the optimized spinal straight line Define the geometric parameters of the spine: midpoint of the spine ,in Unit vector in the spinal direction Vertical unit vector .
[0151] For any patch center coordinates ,calculate:
[0152]
[0153] in, Indicates the longitudinal position along the spine. Indicates the degree of lateral deviation.
[0154] Design a learnable coordinate encoder to enable the network to adaptively learn spatial location information; and integrate spinal coordinates. Mapping to visual features using two layers of MLP Dimensional space: .
[0155] The MLP is configured with an input dimension of 2, a hidden layer dimension of 128, and an output dimension of 768, using the ReLU activation function. Residual connections are used to fuse the features to the original patch. Coordinate embedding is only applied to the foreground patch to avoid background interference.
[0156] S4, Prototype Matching and Classification Steps
[0157] Following a case-based reasoning approach, for each category study An interpretable prototype.
[0158] Category Collection Corresponding to normal (Cobb angle) ), mild ( – ), moderate ( – ) and severe ( The severity is classified into four levels, which is consistent with the SOSORT clinical management guidelines.
[0159] In a specific embodiment, each type is set There are 1 prototype, total There are prototype vectors; all prototypes are uniformly numbered as follows: Each prototype vector has a dimension of Record categories The corresponding prototype index set is (For example, category 0 corresponds to) ).
[0160] For patch features with fused coordinate encoding Calculate the cosine similarity between each patch and each prototype:
[0161]
[0162] in Foreground patch index, For prototype index. All foreground patches and prototypes. cosine similarity set The activation sequence that constitutes this prototype.
[0163] Perform global max pooling only on the foreground patch to obtain the activation strength of each prototype:
[0164]
[0165] in Foreground patch set.
[0166] The category score is defined as the maximum activation value of all prototypes of that class:
[0167]
[0168] in For category The prototype set.
[0169] The final classification prediction is:
[0170]
[0171] S5, Interpretable Result Generation
[0172] By using the predicted categories output by the model and the activation sequences of each prototype, a visually interpretable screening result is generated.
[0173] This classification mechanism is intuitively interpretable: an input image is categorized into a certain severity level because a local region of it shows high similarity to the prototype of that category. By visualizing the activation heatmap, the prototype sample library and the specific anatomical regions that the model focuses on can be directly observed. Figure 2 The diagram shown is a classification illustration of the prototype matching method used in this invention, and the result is a mild category.
[0174] S6, Model Training Steps
[0175] Design a multi-objective joint loss function to simultaneously optimize classification accuracy, prototype quality, and spatial constraints:
[0176]
[0177] in For standard cross-entropy loss, For prototype learning loss, For the spatial constraint loss of the spinal principal axis, represents the weighting coefficient for spatial constraint loss.
[0178] S6.1, Prototype Learning Loss
[0179] Prototype learning loss Includes clustering loss Separation loss and activation equilibrium loss Specifically:
[0180]
[0181] Clustering loss To promote similarity between sample features and similar prototypes, cosine similarity is used, specifically:
[0182]
[0183] in, For sample index within a batch, Batch size; For the sample Real category The corresponding set of prototype indices For the sample With prototype The maximum cosine similarity.
[0184] Separation loss Inter-class discriminability is enhanced by penalizing samples with high similarity to the prototype of the error class. This is the maximum allowed similarity threshold between a sample and a prototype from a different class. As above, specifically:
[0185]
[0186] Activate Equilibrium Loss To prevent prototype collapse where only a few prototypes dominate the classification, and to ensure that all prototypes of the same type are fully activated to capture diverse morphological patterns within that severity level, specifically:
[0187]
[0188] in, and Samples The maximum and minimum activation values among similar prototypes The maximum allowed activation gap threshold.
[0189] S6.2, Spatial Constraint Loss of the Spinal Principal Axis
[0190] The surface features of AIS (such as shoulder tilt, scapular prominence, and lumbar asymmetry) are distributed on both sides of the spinal axis. To guide the activation of the prototype concentrated in clinically relevant anatomical areas near the spine, a spatial constraint loss along the spinal axis was designed.
[0191] Spinal principal axis spatial constraint loss The calculation method is as follows: An adaptive Gaussian space weight is constructed based on the spinal principal axis.
[0192]
[0193] in, Center point for each patch The vertical distance to the main axis of the spine. Estimated by the second eigenvalue of PCA, i.e. , This is the width scaling factor. A key advantage of this design is its adaptability: patients with more severe scoliosis typically have a wider lateral trunk distribution (…). (larger), therefore the Gaussian weight coverage is automatically expanded; while for subjects with a slender build... Smaller size means a narrower area of attention.
[0194] The loss function consists of two parts: a penalty for foreground region deviation from the principal axis and a penalty for background region activation.
[0195]
[0196] in, and Each is a current sample A set of patch indexes for the foreground and background regions; Current sample Medium prototype In position Positive activation value at ( For the current sample (cosine similarity defined in S4). For the current sample In position The Gaussian space weights at the location. The first term is passed through... The first term penalizes activations in the foreground region that are far from the spine, while the second term directly penalizes activations in the background region. This loss guides the model to learn deformity patterns in the anatomical regions surrounding the spine.
[0197] S6.3 Training Strategy
[0198] To effectively utilize the knowledge of the pre-trained backbone network and stabilize prototype learning, a two-stage training strategy is adopted. The warm-up stage freezes the backbone network parameters, training only the prototype layer and coordinate encoder, allowing the prototype to quickly adapt to the feature distribution. The fine-tuning stage unfreezes the backbone network for end-to-end optimization.
[0199] The above description is only a preferred embodiment of the present invention. It should be noted that those skilled in the art can make several improvements without departing from the principle of the present invention, and these improvements should also be considered within the scope of protection of the present invention.
Claims
1. A method for interpretable screening of scoliosis based on a multi-regional sensing prototype network, characterized in that, Includes the following steps: S1. Acquire standardized back surface images I of the subject, adjust the resolution, divide the data into non-overlapping patches, and extract the visual features of each patch. Composition of feature matrix ; S2, for the visual feature matrix Perform PCA foreground segmentation to obtain a set of foreground patch indices. and background patch index set ; S3, Construct the spine reference axis and index the foreground patch. Normalized spine coordinates are calculated for each patch and encoded. Their respective visual features are then fused to obtain the position-aware features of each foreground patch. S4 sets different categories for scoliosis interpretability screening and learns interpretable prototype vectors for each category. Based on the position-aware features of each foreground patch, it calculates the cosine similarity between the patch and each prototype, extracts the spatial maximum activation intensity to obtain the score for each category, and finally outputs the predicted category of image I. S5 uses the predicted categories and activation sequences of each prototype to generate visual and interpretable screening results; S6. Design a multi-objective joint loss function and use a two-stage training strategy to train the visual Transformer backbone network and update the prototype and coordinate encoder. The two-stage training strategy is as follows: during the warm-up stage, the backbone network parameters are frozen, and only the prototype layer and coordinate encoder are trained; During the fine-tuning phase, the backbone network is unfrozen and end-to-end optimization is performed.
2. The method as described in claim 1, characterized in that, In S1, the feature matrix ;in For the number of patches, Let be the feature dimension; denoted as For the first The visual feature vector of each patch That is, visual feature matrix The OK.
3. The method as described in claim 1, characterized in that, Specifically, S2 is: First, the visual feature matrix Perform principal component analysis on the whole, extract the first principal component, calculate the projection value of the visual features of each patch onto the first principal component and normalize it to the nearest integer. The interval is used to obtain the first principal component projection score for each patch. ; Then, based on the first principal component projection score of each patch, the foreground determination direction of image I is determined: Specifically: calculate the corresponding patch in each of the four corner regions of image I. Local mean, and the corresponding values for all patches The overall mean is used to determine if a local mean is higher than the overall mean. If so, the foreground direction corresponds to a value below a set threshold. Value; otherwise, the foreground direction corresponds to a value higher than the set threshold. value; Next, set a threshold for the projection score. A binary foreground mask is generated by combining the determined foreground direction. ; If the foreground direction corresponds to a lower The value will satisfy < patch corresponding mask Assign a value of 1 if the foreground direction corresponds to a higher value, otherwise assign a value of 0; The value will satisfy patch corresponding mask If assigned a value of 1, otherwise assigned a value of 0; Applying morphological operations to masks Fill isolated gaps inside the foreground and remove edge artifacts from the left and rightmost columns; Finally, based on the mask The value is used to obtain the foreground patch index set. and background patch index set ; ； 。 4. The method as described in claim 3, characterized in that, Specifically, S3 is: S3.1, Extracting the set The center spatial coordinate set of all corresponding foreground patches is obtained, and principal component analysis is performed on the coordinate set. The direction of the first principal component is taken as the initial estimate of the spine principal axis. S3.2, group the central spatial coordinate set by row, calculate the average value of the horizontal coordinates of all foreground patches in each row, and combine it with the vertical coordinates of each row to form a coordinate pair, thus forming the spine midline point set; S3.3, using the initial estimated parameters of the spinal principal axis as initial values, fits the straight line containing the spinal reference principal axis by minimizing the sum of squared residuals from the set of spinal midline points to the straight line. ; The parameters of the line optimized using an iterative optimization algorithm; S3.4, calculate the normalized spine coordinates for each foreground patch separately. : in, The original projection distance of the patch along the longitudinal direction of the spine: The original projection distance of the patch along the transverse direction of the spine: S3.5, the normalized spinal coordinates of each patch are mapped to the same coordinates as the visual features using a multilayer perceptron. In a 3D space, each coordinate code is generated and fused into the original patch features using residual connections to obtain position-aware features; Normalized spine coordinates of the nth patch Generated coordinate codes fused into the original patch features To obtain location-aware features : .
5. The method as described in claim 4, characterized in that, Specifically, S4 is: First, let the set of categories be... These correspond to four levels: normal, mild, moderate, and severe. Then, for each category study An interpretable prototype vector ,common Each prototype is uniquely numbered as follows: ; Next, the position-aware features of the nth foreground patch are analyzed. The cosine similarity with each prototype is calculated as follows: All foreground patches and prototypes cosine similarity set The activation sequence that constitutes this prototype; Extract the maximum value from the activation sequence as the activation strength of each prototype: ; Based on the activation intensity of each prototype, calculate the maximum activation value of the prototype belonging to each category: ; Finally, the predicted category of the current input image is determined based on the maximum activation value of each category: For category The corresponding set of prototype indices.
6. The method as described in claim 5, characterized in that, In S5, the interpretable screening results include the following three parts: Output the severity rating results, i.e., based on Determine the final prediction category and output the confidence score for each category. ; Generate interpretable activation heatmaps and extract predicted categories. The activation sequence corresponding to the prototype is summed or maximized. After zero-padding at the corresponding positions of the background region patch in the original mesh, the sequence is reshaped into... The spatial distribution map is then bilinearly upsampled to the original image resolution to obtain a visual activation heatmap superimposed on the original back image I. Retrieve the most similar prototype sample: for the prediction results The prototype with the highest activation intensity The historical samples with the highest activation intensity of the prototype are retrieved from the training set as reference cases to provide intuitive pathological references based on the similarity of local morphological features.
7. The method as described in claim 6, characterized in that, S6 includes: First, a training sample set is constructed, containing several subject body surface images and their corresponding real-world screening category labels of varying degrees. In each training iteration, a sample of [number] samples is extracted from the training sample set. The training samples form a batch input to the backbone network for processing; The loss function is as follows: in, Cross-entropy is used for classification loss; Prototype learning loss; Loss due to spatial constraint of the spinal principal axis; These are the weighting coefficients for the spatial constraint loss; Prototype learning loss Includes clustering loss Separation loss and activation equilibrium loss Specifically: , , These are the weighting coefficients corresponding to each type of loss; Clustering loss Specifically: ; in, For sample index within a batch; For the sample Real category labels The corresponding set of prototype indices For the sample With prototype Maximum cosine similarity; Separation loss Specifically: ; in This is the maximum allowed similarity threshold between a sample and a prototype from a different class. Activate Equilibrium Loss Specifically: ; in, and Samples Maximum and minimum activation strengths among similar prototypes The maximum allowed activation gap threshold; Spinal principal axis spatial constraint loss The calculation method is as follows: An adaptive Gaussian space weight is constructed based on the spinal principal axis. in, Center point for each patch The vertical distance to the main axis of the spine. Estimated by the second eigenvalue of PCA, This is the width scaling factor; The loss function consists of two parts: a penalty for foreground region deviation from the principal axis and a penalty for background region activation. in, and Each is a current sample A set of patch indexes for the foreground and background regions; For the current sample Medium prototype In position Positive activation value at that location, For the current sample Cosine similarity; For the current sample In position Gaussian space weights at the location.