A method and system for predicting the hardness of kidney stones based on cross-modal aligned knowledge distillation

Abstract

This invention discloses a method and system for predicting kidney stone hardness based on cross-modal aligned knowledge distillation, comprising: training a teacher network using multi-channel DECT data, enabling the teacher network to predict stone hardness and extract the physical attribute vector Φ. DECT ;Φ DECT The dimension is d p Each dimension corresponds to a physical attribute that plays a decisive role in hardness prediction; a student network is constructed, which, after receiving a single-channel SECT image, first generates a latent physical attribute vector and then performs hardness prediction based on the latent physical attribute vector; an alignment loss is constructed, which, during the training of the student model, allows the distribution and geometry of to approximate Φ. DECT The SECT image to be predicted is input into the trained student model to predict the hardness of the kidney stone. This invention can transfer rich information from dual-energy CT to a plain CT model to improve the predictive ability of plain CT for kidney stone hardness.

Description

Technical Field

[0001] This invention belongs to the field of medical artificial intelligence, and in particular relates to a method and system for predicting the hardness of kidney stones based on cross-modal aligned knowledge distillation. Background Technology

[0002] Kidney stones are a common disease of the urinary system. The choice of treatment options, such as medication to expel stones, extracorporeal shock wave lithotripsy, and percutaneous nephrolithotomy, is closely related to the hardness of the stones.

[0003] Currently, determining the hardness of kidney stones relies primarily on physician experience and limited imaging features, making it highly subjective and inaccurate. While invasive testing can accurately measure hardness, it causes pain and risks to patients and cannot provide timely information for treatment planning beforehand.

[0004] Dual-energy CT (DECT) provides richer information than conventional CT (SECT). Based on the differences in tissue absorption of X-rays at different energies, it can obtain images specific to various substances, helping to more accurately determine the hardness of stones. However, the high cost and large radiation dose of dual-energy CT scans limit its widespread application.

[0005] Plain CT scans are more commonly used, offering advantages such as low cost and minimal radiation, but they provide relatively limited information. Chinese patent document CN11424093A discloses a method and system for detecting kidney stones based on CT slices; Chinese patent document CN117830223A discloses a method and device for detecting and evaluating kidney stones based on plain CT images. However, most current patent documents rely on CT images for kidney stone detection and cannot predict the hardness of kidney stones. Summary of the Invention

[0006] This invention provides a method and system for predicting the hardness of kidney stones based on cross-modal alignment knowledge distillation, which can transfer the rich information of dual-energy CT to the plain CT model to improve the predictive ability of plain CT for the hardness of kidney stones.

[0007] A method for predicting kidney stone hardness based on cross-modal aligned knowledge distillation includes:

[0008] Training a teacher network using multi-channel DECT data Enable teacher network Predicting stone hardness and extracting physical property vector Φ DECT ;Φ DECT The dimension is d p Each dimension corresponds to a physical property that plays a decisive role in hardness prediction;

[0009] Building a student network The student network After receiving a single-channel SECT image, a latent physical property vector is first generated. Then based on the latent physical property vector Perform hardness prediction;

[0010] Construct an alignment loss and, during the training of the student model, let The distribution and geometry approximate Φ DECT ;

[0011] The SECT image to be predicted is input into the trained student model to predict the hardness of the stone.

[0012] Furthermore, teacher networks The structure is a 3D-ResNet, whose encoder part output is designed as a physical property branch and a residual feature branch;

[0013] The physical property branch is used to output the physical property vector Φ. DECT Φ DECT dimensional d p =3, representing an abstract parameter for the average effective atomic number, average electron density, and composition of the substance, respectively;

[0014] The residual feature branch is used to capture other information that helps in hardness prediction, including texture and mentality; its output is the residual feature vector R. DECT ;

[0015] Teacher Network Final hardness prediction probability It is the physical property vector Φ DECT and residual eigenvector R DECT After feature fusion, it is then classified using a classification head. The resulting formula is:

[0016]

[0017] Furthermore, the training loss of teacher networks It consists of two parts: classification cross-entropy loss L. CE And an optional physical property supervision loss L phy :

[0018]

[0019] Where y is the actual hardness label, and λ phy This indicates that it is a hyperparameter, that is, a manually set weighting coefficient, whose function is to balance the "classification loss" L. CE and "physical attribute supervised loss L" phy Its importance in the total loss of teachers' networks; Φtrue Represents the actual physical property vector.

[0020] Furthermore, the aforementioned student network Includes encoder and causal inference engine

[0021] encoder First from the SECT image Extract a set of basic features F base Then, these basic features are fed into the causal inference engine. The engine's architecture utilizes the decoder portion of a variational autoencoder (VAE), aiming to learn a mapping from SECT features to the distribution of underlying physical properties; causal inference engine Receive basic feature F base Then, output the mean. Sum of logarithmic variance

[0022]

[0023] Then, the latent physical property vector is obtained by sampling using the reparameterization technique.

[0024]

[0025] in, In terms of dimensions, it corresponds to the Φ of the teacher network. DECT Completely identical; ⊙ represents element-wise multiplication, ∈ represents a random noise vector sampled from a standard normal distribution. This represents the standard deviation vector predicted by the student network. This represents a standard multidimensional normal distribution.

[0026] Furthermore, the student network It also includes a residual feature branch for extracting the residual feature vector R of the SECT image. SECT ;

[0027] Student Network Hardness prediction probability It is the physical property vector and residual eigenvector R SECT After feature fusion, it is then classified using a classification head. The resulting formula is:

[0028]

[0029] Furthermore, construct the alignment loss L MA The formula is as follows:

[0030] L MA =w1L dist +w2L geom +w3L causal

[0031] In the formula, L dist For the distribution alignment loss, L geom For local alignment loss, L causal is the causal alignment loss; w1, w2, and w3 are weighting coefficients.

[0032] Calculate the distribution alignment loss L dist When calculating the distance, the Wasserstein distance is used, and the Sinkhorn algorithm is employed.

[0033] Calculate the local alignment loss L geom The formula is as follows:

[0034]

[0035] In the formula, N represents the number of samples in a batch; Indicating teacher network The cosine similarity matrix between the generated physical attribute vectors; Indicates student network The cosine similarity matrix between the generated latent physical property vectors.

[0036] Calculate the causal alignment loss L causal The formula is as follows:

[0037]

[0038] In the formula, N represents the number of samples in a batch, and D KL Denotes the KL divergence; for a sample i, the physical attribute vector Φ is calculated in the teacher network. DECT The influence value of each dimension on hardness prediction is normalized to obtain a causal importance vector. Potential physical properties in student networks Its own prediction results The causal importance vector.

[0039] A kidney stone hardness prediction system based on cross-modal aligned knowledge distillation includes a memory and one or more processors. The memory stores executable code, and when the one or more processors execute the executable code, they implement the aforementioned kidney stone hardness prediction method.

[0040] Compared with the prior art, the present invention has the following beneficial effects:

[0041] 1. This invention achieves accurate hardness prediction with low cost and low radiation, enhancing clinical applicability. Through knowledge distillation, it transfers the precise analytical capabilities of expensive and high-radiation-dose dual-energy CT (DECT) to the widely used, low-cost, and low-radiation-dose plain CT (SECT). This allows for high-precision kidney stone hardness prediction using only SECT images in routine examinations, significantly reducing medical costs and patients' radiation exposure risks.

[0042] 2. Improved model interpretability and reliability. Traditional deep learning models are often "black boxes," with decision-making processes difficult to understand. This invention designs a teacher network and a forced student network with physical property decoupling capabilities to infer latent physical properties, thus associating the model's prediction process with explicit physical meanings (such as effective atomic number and electron density). This design not only makes it easier for doctors to understand and trust the model's predictions, but also provides a clear path for analysis and tracing the problem when predictions deviate, enhancing the model's reliability.

[0043] 3. Enhanced model generalization ability and adaptability to data changes. Because this invention aligns with more fundamental and stable physical properties, rather than superficial image features that are easily affected by scanning parameters and equipment models, the trained student model exhibits better generalization ability to new and unseen data. Even if the input SECT images differ in brightness, contrast, etc., as long as the intrinsic physical properties of the stones remain unchanged, the model can still make robust hardness predictions, which is crucial for deployment and application across different hospitals and equipment. Attached Figure Description

[0044] Figure 1 This is a flowchart of a method for predicting the hardness of kidney stones based on cross-modal aligned knowledge distillation, according to an embodiment of the present invention.

[0045] Figure 2 This is a structural diagram of the teacher network in an embodiment of the present invention.

[0046] Figure 3 This is a structural diagram of the student network in an embodiment of the present invention. Detailed Implementation

[0047] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be noted that the embodiments described below are intended to facilitate the understanding of the present invention and do not constitute any limitation thereof.

[0048] The core of traditional knowledge distillation (KD) lies in having the student network mimic the behavior of the teacher network, typically manifested in soft labels at the output layer or feature maps at intermediate layers. However, in cross-modal tasks from DECT to SECT, this mimicry has fundamental flaws. The teacher network is powerful because its inputs (DECT data, such as multi-energy virtual monoenergetic images (VMI), effective atomic number (Z_eff) maps, electron density (ρ_e) maps, etc.) themselves contain rich physical information. In contrast, the student network's input (SECT) lacks a physical dimension. Therefore, this invention does not require the student network to "copy" the teacher network's features, but rather requires the student network to "infer" the latent physical properties related to hardness prediction from the SECT data, and align these inferred properties with the real physical properties extracted by the teacher network from the DECT in a shared, structured latent space.

[0049] The core assumption of this invention is that the hardness (H) of kidney stones is determined by their intrinsic physicochemical composition. DECT can directly or indirectly measure these physical properties (referred to in this invention as explicit physical characteristic vectors Φ) through multi-energy imaging. DECT Examples of these include effective atomic number and electron density. Meanwhile, the CT value (Hounsfield Unit, HU) and texture features (referred to as appearance features X in this invention) in SECT images... SECT () is a projection and hybrid manifestation of these intrinsic physical properties on a low-dimensional information space. Therefore, there exists a […] from X SECT To Φ DECT The highly nonlinear mapping relationship. The goal of this invention is to construct a student model that can not only obtain X SECT Predicting hardness H allows for the explicit construction of an intermediate representation—a vector of latent physical properties—in the process. This vector is designed to be related to the real physical property Φ DECT The best estimate. The core of the distillation process then becomes alignment. and Φ DECT Instead of aligning generic, uninterpretable CNN feature maps, a triple alignment loss is calculated during cross-modal knowledge distillation. The updated student network is then deployed as a lightweight prediction model, taking a new plain CT scan as input and outputting a hardness prediction. The specific process is as follows: Figure 1 As shown.

[0050] The framework of the system of this invention comprises three key parts:

[0051] Teacher Network (T): A deep network trained on DECT data with decoupling capabilities for physical properties. It can not only predict stone hardness but also explicitly extract defined physical property vectors Φ within its network. DECT .

[0052] Student Network (S): It receives SECT input, and its internal structure is designed to first generate a vector of latent physical properties. Then, hardness prediction is performed based on this vector.

[0053] Alignment Module (M): This is the bridge of distillation. Instead of using simple L2 distance or KL divergence to match features, it ensures that the latent physical property vectors inferred by the student are... and the real physical attribute vector Φ extracted by the teacher DECT They have similar geometric structures, distribution characteristics, and causal relationships in a high-dimensional space.

[0054] The input to the teacher network T is multi-channel DECT data, one data sample. It is a stacked tensor containing virtual monoenergetic images (VMIs), effective atomic number maps (Z_eff maps), electron density maps, etc., at different keV. Assume the input contains C... T One channel.

[0055]

[0056] Teacher Network The structure is a 3D-ResNet. For example... Figure 2 As shown, its encoder section The output is designed with two branches:

[0057] Physics Branch: The output of this branch is the explicit physical property vector Φ defined in this invention. DECT The dimension d of this vector p It is predefined, and each dimension corresponds to a physical quantity that plays a decisive role in hardness. Define d. p =3, representing an abstract parameter for the average effective atomic number, average electron density, and material composition, respectively. This part of the network is trained using a specific loss function (which may require additional prior physical knowledge or auxiliary tasks) to ensure that its output is indeed highly correlated with the physical measurements in the DECT images. Its output is:

[0058]

[0059] Residual Branch: This branch captures complex, unstructured information (such as texture and morphology) that, in addition to the defined physical properties, still contributes to hardness prediction. Its output is the residual feature vector R. DECT .

[0060]

[0061] Final hardness prediction probability of teacher network It is formed by fusing the features of these two branches and then passing them through a classification head. Obtained.

[0062]

[0063] Where [·;·] represents vector concatenation.

[0064] Training loss in teacher networks It consists of two parts: the standard classification cross-entropy loss L CE And an optional physical property supervision loss L phy (If the target physical values ​​can be calculated directly from the DECT image).

[0065]

[0066] Where y is the actual hardness label. This design ensures Φ DECT It has a clear physical meaning.

[0067] like Figure 3 As shown, student network The input is a single-channel plain CT image. Its core innovation lies in its causal inference engine. The goal of this engine is to "recover" the underlying physical properties from the apparent features of the SECT. (Student network encoder) Firstly, from Extract a set of basic features These basic characteristics are then fed into causal inference. The engine's architecture utilizes the decoder portion of a variational autoencoder (VAE), aiming to learn a mapping from SECT features to the distribution of latent physical properties. Specifically, this invention forces it to output parameters of a Gaussian distribution, i.e., the mean. Sum of logarithmic variance

[0068]

[0069] Then, the latent physical property vector is obtained by sampling using the reparameterization trick.

[0070]

[0071] this In terms of dimensions, it corresponds to the Φ of the teacher network. DEcT Totally consistent.

[0072] Similar to the teacher network, the student network also has a residual feature branch, used to extract SECT-specific information R that cannot be interpreted by physical properties. SECT .

[0073]

[0074] Finally, the probability of predicting the hardness of the student network. It is determined by the physical properties and residual characteristics inferred from it:

[0075]

[0076] This design, through structural constraints, requires the student network to first form a "belief" about physical properties (i.e., Only after understanding the distribution of the data can a final judgment be made.

[0077] Alignment Module It is a connection and The bridge. This invention utilizes an alignment loss L comprising multiple components. MA Its goal is to make The distribution and geometry of Φ are infinitely close. DECT .

[0078] L MA =w1L dist +w2L geom +w3L causal

[0079] 1. Distribution Alignment Loss (L) dist )

[0080] This invention uses Wasserstein distance and calculates it using the Sinkhorn algorithm. For a batch of data, there are two sets of vectors. and

[0081]

[0082] in Let Lip1 represent the empirical distribution, and Lip1 represent the set of 1-Lipschitz functions. In practice, this distance is approximated by an auxiliary discriminator network using the Maximum Mean Discrepancy (MMD), which measures the distance between the two distributions in a reproducing kernel Hilbert space (RKHS).

[0083]

[0084] Where k(·,·) is a kernel function, i.e., a Gaussian kernel. MMD loss can effectively narrow the mean, variance, and even higher-order moments of two distributions.

[0085] 2. Local alignment loss (L) geom )

[0086] Distribution alignment guarantees global properties, but it's also necessary to preserve the local neighborhood relationships between data points. That is, if two stones are similar in the physical attribute space of DECT, then they should also be similar in the latent physical attribute space inferred by SECT. This invention achieves this by constructing a relation kernel matrix. For N samples in a batch, the cosine similarity matrix between the physical attribute vectors of the teacher network is first calculated.

[0087]

[0088] Similarly, the similarity matrix of the latent physical attribute vectors inferred from the student network is calculated.

[0089]

[0090] The geometric alignment loss is simply the Frobenius norm of these two relation matrices:

[0091]

[0092] This loss term forces the student network to learn an embedding space that preserves the relative relationships between samples.

[0093] 3. Causal alignment loss (L causal )

[0094] In addition to aligning the physical attribute vectors themselves, it is also necessary to align the strength of the causal relationship between these physical attributes and the final prediction result (hardness). In a teacher network, this invention can approximate the contribution of each physical attribute dimension to the final decision by analyzing gradients. For a sample i, physical attribute Φ DECTThe k-th dimension predicts hardness (in logit) The influence of (for example) can be approximated as the magnitude of the gradient:

[0095]

[0096] This invention normalizes these influence values ​​to obtain a causal importance vector. in

[0097]

[0098] Similarly, this invention can compute latent physical properties in student networks. Its own prediction results Causal importance vector

[0099]

[0100] The goal of causal alignment loss is to make the distributions of the two causal importance vectors as consistent as possible, which is measured using KL divergence.

[0101]

[0102] This loss does not concern itself with the specific numerical values ​​of physical properties, but rather with the decision-making logic of "which physical property is more important." For example, if the teacher network primarily relies on "effective atomic number" to determine hardness, then this loss will force the student network to also learn to primarily rely on its inferred "potential effective atomic number" to make judgments, thereby achieving knowledge transfer at the decision-making logic level.

[0103] The complete training process is divided into two stages:

[0104] Phase 1: Training Teacher Networks Using the DECT dataset Minimize the loss function of the teacher network

[0105]

[0106] After training, the parameters of the teacher network It has been frozen.

[0107] Phase Two: Training Student Networks via LPD-CMA Distillation Using paired DECT-SECT datasets For each data triple, perform the following calculation:

[0108] 1. Forward propagation: Enter frozen teacher network Hardness prediction and explicit physical property vectors Will Enter student network Hardness prediction and potential physical property vectors

[0109] 2. Calculate the total loss using backpropagation: Calculate the total loss L Total Student network parameters The gradient, and update Note that the parameters of the teacher network are never updated.

[0110] The embodiments described above provide a detailed explanation of the technical solutions and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, additions, and equivalent substitutions made within the scope of the principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A kidney stone hardness prediction method based on cross-modal alignment knowledge distillation, characterized in that, include: Training a teacher network with multi-channel DECT data , making the teacher network predict stone hardness and extract a physical property vector ; The dimension of the physical property vector is , each dimension corresponds to a physical property that plays a decisive role in hardness prediction. Teacher Network The structure is a 3D-ResNet, whose encoder output is designed as a physical property branch and a residual feature branch; the physical property branch is used to output physical property vectors. , Dimensions , representing abstract parameters of average effective atomic number, average electron density, and material composition, respectively; the residual feature branch is used to capture other information helpful for hardness prediction, including texture and morphology, and its output is a residual feature vector. Teacher Network Final hardness prediction probability It is the physical property vector and residual eigenvectors After feature fusion, it is then classified using a classification head. The resulting formula is: Building a student network The student network After receiving a single-channel SECT image, a latent physical property vector is first generated. Then based on the potential physical property vector Perform hardness prediction; The student network Includes encoder and causal inference engine encoder First from the SECT image Extract a set of basic features Then, these basic features are fed into the causal inference engine. The engine's architecture uses the decoder portion of a variational autoencoder (VAE), aiming to learn a mapping from SECT features to the distribution of underlying physical properties; causal inference engine Receive basic features Then, output the mean. Sum of logarithmic variance : Then, the latent physical property vector is obtained by sampling using the reparameterization technique. : in, In terms of dimensions, it is related to the teacher network. completely consistent; This represents element-wise multiplication. Let represent a random noise vector sampled from a standard normal distribution. This represents the standard deviation vector predicted by the student network. Represents a standard multidimensional normal distribution; The student network It also includes a residual feature branch for extracting the residual feature vector of the SECT image. Student Network Hardness prediction probability It is the physical property vector and residual eigenvectors After feature fusion, it is then classified using a classification head. The resulting formula is: Construct an alignment loss and, during the training of the student model, let The distribution and geometric structure approximate ; The SECT image to be predicted is input into the trained student model to predict the hardness of the stone.

2. The method for predicting kidney stone hardness based on cross-modal aligned knowledge distillation according to claim 1, characterized in that, Training loss in teacher networks It consists of two parts: classification cross-entropy loss and classification cross-entropy loss. And an optional physical property supervised loss : in, It's a genuine hardness label. Indicates the weighting coefficient. Represents the actual physical property vector.

3. The method for predicting kidney stone hardness based on cross-modal aligned knowledge distillation according to claim 1, characterized in that, Constructing alignment loss The formula is as follows: In the formula, For distribution alignment loss, For local alignment loss, For causal alignment loss; , and These are the weighting coefficients.

4. The method for predicting kidney stone hardness based on cross-modal aligned knowledge distillation according to claim 3, characterized in that, Calculate distribution alignment loss When calculating the distance, the Wasserstein distance is used, and the Sinkhorn algorithm is employed.

5. The method for predicting kidney stone hardness based on cross-modal aligned knowledge distillation according to claim 3, characterized in that, Calculate local alignment loss The formula is as follows: In the formula, This indicates the number of samples in a batch; , indicating teacher network The cosine similarity matrix between the generated physical attribute vectors; Represents student network The cosine similarity matrix between the generated latent physical property vectors.

6. The method for predicting kidney stone hardness based on cross-modal aligned knowledge distillation according to claim 3, characterized in that, Calculate causal alignment loss The formula is as follows: In the formula, This indicates the number of samples in a batch. Denotes the KL divergence; for a sample Calculate physical attribute vectors in the teacher network The influence value of each dimension on hardness prediction is normalized to obtain a causal importance vector. ; Potential physical properties in student networks Its own prediction results The causal importance vector.

7. A kidney stone hardness prediction system based on cross-modal aligned knowledge distillation, characterized in that, The device includes a memory and one or more processors, wherein the memory stores executable code, and the one or more processors execute the executable code to implement the kidney stone hardness prediction method according to any one of claims 1-6.

Images