Knee osteoarthritis image prediction method and system based on diffusion model

Abstract

The application discloses a knee osteoarthritis image prediction method and system based on a diffusion model. The method comprises the following steps: acquiring multi-modal data of a patient in a baseline period, wherein the multi-modal data comprises X-ray images, MRI sequences and clinical features; adopting a latent diffusion model as a basic prediction network architecture; adding a multi-modal fusion module and a cross-attention mask regulation in a U-Net structure of the latent diffusion model; selecting an AdamW optimizer; performing end-to-end training on the model; inputting the multi-modal data into the trained latent diffusion model; completing feature fusion through adaptive gating; and performing inverse diffusion denoising under the constraint of an inflation joint gap key point mask, and finally decoding and outputting a predicted knee X-ray image. The application can improve the accuracy of joint image prediction.

Description

Technical Field

[0001] This invention generally relates to the field of artificial intelligence technology. More specifically, this invention relates to a method and system for image prediction of knee osteoarthritis based on a diffusion model. Background Technology

[0002] Knee osteoarthritis (KOA) is a highly prevalent degenerative joint disease characterized by knee pain and swelling as the initial symptoms, ultimately leading to joint deformity and disability. It is one of the leading causes of chronic pain and disability worldwide. The progression of KOA is irreversible, and there is currently no effective treatment to reverse its progression. In the end stage of the disease, total knee replacement is the only treatment option. Therefore, early and accurate identification of high-risk individuals and personalized, targeted clinical interventions are of significant clinical importance for improving patient prognosis. Currently, clinicians primarily rely on subjective visual assessment of knee X-ray or MRI images to monitor structural changes such as joint space narrowing, osteophyte formation, articular surface irregularities, and subchondral bone sclerosis. In terms of prediction, standard clinical protocols and existing artificial intelligence (AI) prediction models mainly focus on predicting non-radiographic outcomes, such as changes in the Kellgren-Lawrence (KL) grade, pain scores, or the probability of future total knee replacement—semi-quantitative or categorical indicators. In the field of image generation and prediction, there are currently some AI technologies based on image-to-image generation (such as the StyleGAN-based latent extrapolation method SLE, and the Image Consistency Radiation Diffusion Network IC-RDN) used to generate future follow-up X-ray images.

[0003] However, current diagnostic methods for knee osteoarthritis (KOA) have the following shortcomings: 1. Oversimplification and poor interpretability of single indicators: Existing predictive models often output single-dimensional scores or probabilities such as the KL grade. These semi-quantitative indicators oversimplify the spatial complexity of KOA progression. These single values ​​cannot intuitively reflect key structural details such as asymmetric joint space narrowing, degeneration of specific compartments, and osteophyte morphology, resulting in a lack of clinical visualization and interpretability, making it difficult to assist doctors in developing personalized surgical or intervention plans. 2. High subjectivity and poor consistency: Diagnosis and prediction based on doctors' visual assessment or KL grade involve significant inter-observer variability, severely limiting the objectivity of current predictive tools. 3. Insufficient utilization of multimodal clinical prior knowledge: Most existing X-ray image generation models only use a single image modality as input, ignoring the multimodal clinical prior knowledge of individual patients. The pathogenesis of KOA is complex, and the patient's demographic characteristics (such as age, sex, BMI) and the soft tissue pathological features captured by MRI, such as the meniscus, synovium, and fat pad, are key risk factors affecting KOA progression. The lack of fusion of this information leads to predictions that deviate from individualized clinical realities. 4. Insufficient focus on lesion areas and feature distortion: Existing image generation networks (such as StyleGAN or general diffusion models) lack constraints on specific anatomical regions during the generation process, often dispersing the main computational power into irrelevant background noise. This results in the model's inability to effectively capture the areas most clinically significant for KOA progression (i.e., around the joint space), and the generated predicted images are prone to spatial inconsistencies and distortions in key clinical features such as osteophyte morphology and joint alignment. Summary of the Invention

[0004] To address at least some of the shortcomings of the aforementioned prediction methods, such as outputting only a single score, lacking intuitive visualization features, and failing to fully integrate multimodal clinical information, this invention proposes a diffusion model-based image prediction method and system for knee osteoarthritis (KOA). This aims to achieve automatic, accurate, and clinically relevant future KOA image prediction, assisting physicians in individualized risk assessment and early intervention decisions. Therefore, this invention provides solutions in the following aspects. The first aspect of the present invention provides an image prediction method for knee osteoarthritis based on a diffusion model, comprising: acquiring multimodal data of a patient at baseline, the multimodal data including X-ray images, MRI sequences, and clinical features; employing a latent diffusion model as the basic prediction network architecture, adding a multimodal fusion module and cross-attention mask modulation to the U-Net structure of the latent diffusion model; selecting AdamW as the optimizer; performing end-to-end training on the model; inputting the multimodal data into the trained latent diffusion model, performing feature fusion through adaptive gating, and performing inverse diffusion denoising under the constraint of a key point mask for the expanded joint space, and finally decoding and outputting the predicted knee joint X-ray image.

[0005] In one embodiment, the latent diffusion model first utilizes an image encoder with a variational autoencoder (VAE). By compressing real high-dimensional X-ray images into a low-dimensional latent feature space, latent features are obtained. The forward diffusion process gradually adds Gaussian noise to the potential features at time step [number missing]. The noise addition formula is: ,in It is the cumulative product term of the diffusion scalar; the inverse denoising process is performed by a U-Net network parameterized by the model, which is based on the latent representation of the baseline X-ray image. Guided by multimodal conditional information C, noise is predicted and eliminated; the loss function for model training is the mean squared error (MSE). ; After denoising, it is processed by an image decoder. Features are restored to a high-resolution image. .

[0006] In one embodiment, the multimodal fusion module uses a feature extraction network. Extract the first Features of an MRI slice Using a clinical embedding layer Extracting clinical features The fusion features are calculated using an adaptive gating mechanism, and the calculation formula is as follows: ,in, This represents a gating unit used to calculate the dynamic weighting coefficients when clinical features interact with MRI features. Indicates the fused first A multimodal feature vector, after being augmented with positional encoding, is used as a conditional key and value matrix input to the cross-attention layer of the diffusion model.

[0007] In one embodiment, the cross-attention mask modulation includes: locating and extracting key points of the joint space using anatomical landmarks, generating a two-dimensional dilated mask matrix M, and adjusting its size to match the feature map size of the U-Net; in the cross-attention layer, redistributing spatial weights using the mask; and setting... Define the spatial weight matrix for the original cross-attention feature map: ,in, and These are the trainable, independent weight parameters assigned by the system to the inside of the mask (critical lesion area) and the outside of the mask (non-critical background area); the final feature update formula for the l-th layer inside the U-Net is: ,in This indicates element-wise multiplication.

[0008] In one embodiment, the method further includes directly quantifying the predictive accuracy of anatomical degeneration using the Joint Interarticular Distance Error Formula (JSDE), which is used to locate and calculate six key pairs of points between the femur and tibia that best characterize the anatomical structure: ; in, This represents the coordinates of the k-th pair of anatomical key points in the real image x. Represents the generated predicted image The corresponding k-th pair of keypoint coordinates, This formula calculates the Euclidean distance between two points; it also calculates the relative proportional error of the joint gap width between the generated image and the real image.

[0009] A second aspect of the present invention provides an image prediction system for knee osteoarthritis based on a diffusion model, comprising: a data acquisition module for acquiring multimodal data of a patient at baseline, the multimodal data including X-ray images, MRI sequences, and clinical features; a model training module for using a latent diffusion model as the basic prediction network architecture, adding a multimodal fusion module and cross-attention mask modulation to the U-Net structure of the latent diffusion model; selecting AdamW as the optimizer; performing end-to-end training on the model; and an image prediction module for inputting the multimodal data into the trained latent diffusion model, performing feature fusion through adaptive gating, and performing inverse diffusion denoising under the constraint of a key point mask for the expanded joint space, and finally decoding and outputting the predicted knee X-ray image.

[0010] In one embodiment, the latent diffusion model first utilizes an image encoder with a variational autoencoder (VAE). By compressing real high-dimensional X-ray images into a low-dimensional latent feature space, latent features are obtained. The forward diffusion process gradually adds Gaussian noise to the potential features at time step [number missing]. The noise addition formula is: ,in It is the cumulative product term of the diffusion scalar; the inverse denoising process is performed by a U-Net network parameterized by the model, which is based on the latent representation of the baseline X-ray image. Guided by multimodal conditional information C, noise is predicted and eliminated; the loss function for model training is the mean squared error (MSE). ; After denoising, it is processed by an image decoder. Features are restored to a high-resolution image. .

[0011] In one embodiment, the multimodal fusion module uses a feature extraction network. Extract the first Features of an MRI slice Using a clinical embedding layer Extracting clinical features The fusion features are calculated using an adaptive gating mechanism, and the calculation formula is as follows: ,in, This represents a gating unit used to calculate the dynamic weighting coefficients when clinical features interact with MRI features. Indicates the fused first A multimodal feature vector, after being augmented with positional encoding, is used as a conditional key and value matrix input to the cross-attention layer of the diffusion model.

[0012] In one embodiment, the cross-attention mask modulation includes: locating and extracting key points of the joint space using anatomical landmarks, generating a two-dimensional dilated mask matrix M, and adjusting its size to match the feature map size of the U-Net; in the cross-attention layer, redistributing spatial weights using the mask; and setting... Define the spatial weight matrix for the original cross-attention feature map: ,in, and These are the trainable, independent weight parameters assigned by the system to the inside of the mask (critical lesion area) and the outside of the mask (non-critical background area); the final feature update formula for the l-th layer inside the U-Net is: ,in This indicates element-wise multiplication.

[0013] In one embodiment, the method further includes directly quantifying the predictive accuracy of anatomical degeneration using the Joint Interarticular Distance Error Formula (JSDE), which is used to locate and calculate six key pairs of points between the femur and tibia that best characterize the anatomical structure: ; in, This represents the coordinates of the k-th pair of anatomical key points in the real image x. Represents the generated predicted image The corresponding k-th pair of keypoint coordinates, This formula calculates the Euclidean distance between two points; it also calculates the relative proportional error of the joint gap width between the generated image and the real image.

[0014] This invention enables end-to-end image generation, directly outputting X-ray images of the joint at a specific future time point. Compared to existing systems that only output KL grading, this invention preserves the complex spatial topological details of KOA progression (such as asymmetric space narrowing), providing physicians with highly interpretable visual evidence. Attached Figure Description

[0015] The above and other objects, features, and advantages of exemplary embodiments of the present invention will become readily apparent upon reading the following detailed description with reference to the accompanying drawings. In the drawings, several embodiments of the invention are illustrated by way of example and not limitation, and like or corresponding reference numerals denote like or corresponding parts, wherein: Figure 1 This demonstrates an image prediction method for knee osteoarthritis. Figure 2 This is a schematic diagram of an end-to-end system architecture; Figure 3 This shows the prediction results. Detailed Implementation

[0016] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention. It should be understood that the terms "first," "second," "third," and "fourth," etc., in the claims, specification, and drawings of this invention are used to distinguish different objects, rather than to describe a specific order. The terms "comprising" and "including" used in the specification and claims of this invention indicate the presence of the described features, integrals, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or collections thereof. It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used in this specification and claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in this specification and claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes such combinations. As used in this specification and claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if [described condition or event] is detected" may be interpreted, depending on the context, as "once determined," "in response to determination," "once [described condition or event] is detected," or "in response to detection of [described condition or event]." The first aspect of this invention provides a method for risk assessment of the home environment for the elderly. Specific embodiments of the invention will now be described in detail with reference to the accompanying drawings. Figure 1 The diagram shown illustrates the image prediction method for knee osteoarthritis according to the present invention. (See attached diagram.) Figure 1 It can be seen that the image prediction method for knee osteoarthritis of the present invention specifically includes steps S100-S300: Step S100: Acquire multimodal data of the patient at baseline, including X-ray images, MRI sequences, and clinical features; Step S200: Using a latent diffusion model as the basic prediction network architecture, a multimodal fusion module and cross-attention mask modulation are added to the U-Net structure of the latent diffusion model; AdamW is selected as the optimizer; and the model is trained end-to-end. Step S300: Input the multimodal data into the trained latent diffusion model, complete feature fusion through adaptive gating, and perform inverse diffusion denoising under the constraint of the key point mask of the expanded joint space, and finally decode and output the predicted knee joint X-ray image.

[0017] This invention collects multimodal data (including X-ray images, MRI sequences, and clinical features) from patients at baseline and deeply integrates this heterogeneous multimodal information using a feature fusion network with an adaptive gating mechanism. The fused features are then used as conditional inputs into a latent diffusion model. In the crucial step of image generation (the cross-attention layer), this invention innovatively introduces a KOA key feature-guided attention mechanism. This mechanism forces the model to precisely focus on the joint space region most clinically significant for KOA progression, thereby filtering background noise and ultimately decoding and generating future predictive X-ray images that highly reproduce the evolution of disease structures (such as joint space narrowing and osteophyte formation).

[0018] Figure 2 This is a schematic diagram of an end-to-end system architecture, consisting of... Figure 2 The system architecture of the present invention shown mainly includes three basic parts: data input, core generation model, and result evaluation.

[0019] In this invention, the algorithm's input and output modules are used to acquire input data and obtain the predicted image, respectively. Specifically, the model input includes three dimensions of multimodal data: 1. Baseline X-ray image of the patient's knee joint (denoted as...) ).

[0020] 2. Baseline 3D MRI sequence (denoted as...) , where s is the number of sequence slices extracted.

[0021] 3. Structured individual clinical information characteristics (denoted as c, including age, BMI, gender, and race parameters).

[0022] Model output: High-resolution X-ray images of the knee joint predicted at specific future follow-up time points (e.g., after 72 or 96 months).

[0023] Overall Algorithm Model: This invention constructs an end-to-end deep learning generative model based on the Latent Diffusion Model (LDM) architecture, and deeply couples the "adaptive multimodal feature fusion network" and the "anatomical key point mask guidance mechanism".

[0024] Furthermore, the prediction algorithm of the present invention includes the following specific implementation methods for sub-modules: 1. Forward diffusion process and latent diffusion denoising model: In order to reduce computational costs, this invention first utilizes a variational autoencoder (VAE) image encoder. By compressing real high-dimensional X-ray images into a low-dimensional latent feature space, latent features are obtained. The forward diffusion process gradually adds Gaussian noise to the potential features at time step [number missing]. The noise addition formula is: , in It is the cumulative product term of the diffusion scalar. Represents a normal distribution; This represents the image after adding noise at time step t; This represents the initial, clean image without added noise.

[0025] The inverse denoising process is performed by the U-Net network, which is parameterized by the model, based on the latent representation of the baseline X-ray image. Guided by multimodal conditional information C, noise is predicted and eliminated. The loss function for model training is the mean squared error (MSE): .

[0026] in, Represents the model loss function; This represents the expectation under the conditions x0, xb, C, t, which is the mean of multiple samples in actual training; x0 represents the target image. Represents a baseline image; This represents the actual noise at the current time step t; This represents the noise predicted by the model; express Latent variables obtained via VAE; Represents a baseline image The latent variables obtained through VAE; t represents the denoised time step, and C represents clinical information.

[0027] After denoising, it is processed by an image decoder. Features are restored to a high-resolution image. .

[0028] 2. Multi-modal Fusion Module: This invention uses a feature extraction network Extract the first Features of an MRI slice Using clinical embedding layer Extracting clinical features Then, an adaptive gating mechanism is used to calculate the fused features. The calculation formula is as follows: .

[0029] in, This represents a gating unit used to calculate the dynamic weighting coefficients when clinical features interact with MRI features. Indicates the fused first A multimodal feature vector. After adding positional encoding, this feature is used as a conditional key and value matrix input to the cross-attention layer of the diffusion model. This represents the features extracted from the patient's clinical information by the clinical encoder.

[0030] 3. KOA Feature-Targeting Attention Mechanism: This invention extracts key points of the joint space by locating anatomical landmarks, generates a two-dimensional dilated mask matrix M (values ​​of 1 covering the key region and 0 for the background), and adjusts its size to match the feature map size of U-Net. In the cross-attention layer, spatial weights are redistributed using the mask.

[0031] set up Define the spatial weight matrix for the original cross-attention feature map. .

[0032] in and These are the trainable, independent weight parameters assigned by the system to the area inside the mask (critical lesion region) and the area outside the mask (non-critical background region). The final feature update formula for the l-th layer inside the U-Net is: ; in, This indicates element-wise multiplication. This represents the (l+1)th layer of the transformer model; The representation extracted by the encoder of VAE; Represents a multilayer sensing network; This represents the mask after weighting by the weights of the w1 joint region and the w2 background region. This represents the attention weight matrix of layer l; This means that the weights are multiplied onto the attention matrix. This mechanism forces the model to precisely focus on the most clinically significant interarticular region of KOA during generation.

[0033] 4. Evaluation Module: Includes a dual verification system. This invention directly quantifies the accuracy of predicting anatomical degeneration by calculating the Joint Space Distance Error (JSDE).

[0034] .

[0035] The above formula is used to locate and calculate six key pairs of points between the femur and tibia that best characterize the anatomical structure. This represents the coordinates of the k-th pair of anatomical key points in the real image x. Represents the generated predicted image The corresponding k-th pair of keypoint coordinates, This formula calculates the Euclidean distance between two points. It also calculates the relative proportional error of the joint gap width between the generated and ground truth images.

[0036] Furthermore, expert double-blind evaluation can be conducted: clinical experts score and rank the generated images based on image fidelity and key KOA features (such as osteophytes, space narrowing, etc.) while concealing the model's source, thus verifying its clinical usability.

[0037] In one specific embodiment of the present invention, the prediction method provided by the present invention is used to predict knee osteoarthritis images 72 months later. The specific process is as follows: 1. Data preparation: Dataset: Multimodal data from the OAI (Osteoarthritis Initiative) database were selected, and knee joint data of subjects that simultaneously included baseline X-ray, baseline IW-TSE MRI sequence (sagittal plane), clinical parameters (age, sex, BMI, race), and X-ray images at 72 months of follow-up were selected, totaling 5913 cases.

[0038] Data partitioning: Data was randomly divided into training, validation, and test sets in a 4:1:1 ratio. To prevent data leakage, both knees of the same patient could not appear in both the training and validation / test sets simultaneously.

[0039] 2. Operating environment: Hardware and software platform: The experiment was conducted using Python 3.8 and PyTorch 2.1.0 frameworks, and two NVIDIA A40 48GB GPUs were used for model training.

[0040] 3. Implementation steps: Step 1 (Data Preprocessing): The anatomical landmarks from baseline and follow-up X-rays were extracted using the KNEEL system to locate the center of the knee joint, and 120mm×120mm images were cropped and histogram equalization was performed. Eight slices of the central region were extracted from the MRI images.

[0041] Specifically, for the patient's sagittal PDMRI sequence (generally scanning 40-100 slices), normalization and uniform size cropping are first performed. Then, a cartilage segmentation model is used to segment the cartilage of each slice to obtain the cartilage mask region; the windows of the 16 slices with the largest cumulative cartilage area in the MRI sequence are taken, and every other slice is taken to obtain 8 slices.

[0042] Step 2 (Model Configuration and Network Training): The Latent Diffusion Model (LDM) is used as the basic prediction network architecture. This architecture does not directly process the massive pixel-level image; instead, it utilizes a pre-trained image encoder to compress the input baseline X-ray image into a low-dimensional latent feature space. In this embodiment, the forward noise addition and backward denoising timesteps of the diffusion model are set to 1000 steps.

[0043] In terms of specific network structure, the diffusion model uses a U-Net structure as the core module for performing the denoising task. U-Net is a classic symmetrical "U"-shaped deep learning network that processes image features through encoding (downsampling to extract global features) and decoding (upsampling to restore local details). To add an attention mask to U-Net, the system first extracts and connects key points at the edge of the knee joint space using an anatomical landmark localization algorithm, and then performs morphological dilation to generate a two-dimensional spatial mask that accurately covers the joint space region. In the cross-attention layer inside U-Net, the system uses this mask as a weight control matrix. The region inside the mask (joint space region) and the region outside the mask (background region) are assigned two independent sets of trainable weights, which are multiplied by the attention weight map. This operation forces the U-Net model to prioritize focusing on pathologically relevant regions when extracting features.

[0044] For model optimization, the system uses the mean squared error (MSE) of noise prediction as the loss function. To promote fast and stable convergence of the network, the optimizer selects the AdamW algorithm, which can adaptively adjust the parameter update step size to effectively reduce the error between predicted noise and real noise. The specific hyperparameter configuration for training is as follows: the batch size is set to 36, the initial learning rate is set to 5e-6, and a learning rate decay strategy is set to smoothly decay to 5e-7 during the 10,000-step training process. The entire model (including the multimodal fusion module and the diffusion network) is jointly trained end-to-end for a total of 300 epochs.

[0045] Step 3 (Multimodal Feature Fusion and Backdiffusion Prediction): Input the baseline X-ray latent representation of the test set patients, the baseline MRI sequence (extracting the central 8 slices), and structured clinical information (such as age, BMI, etc.) into the trained model.

[0046] The specific implementation of feature fusion: The system first extracts the 3D spatial feature sequence of MRI and the semantic feature vector of clinical information through a feature extraction network. Then, an adaptive gating mechanism is used for fusion: the gating unit calculates the interaction between clinical semantic features and MRI spatial features, outputting a dynamic weight coefficient. This weight coefficient is used to scale the clinical information and superimpose it onto the MRI features position by position, thus naturally integrating heterogeneous clinical information into the image features, forming a unified multimodal fusion feature. This feature is then further encoded with positional information and input into the cross-attention layer of the U-Net.

[0047] The specific implementation of inverse diffusion: When generating the target image, the system first initializes a state in the latent feature space consisting entirely of random Gaussian noise. Guided by the aforementioned multimodal fusion features and under the spatial constraints of the aforementioned "expanded joint gap mask," the model performs a reverse denoising process. Based on the learned patterns from training, the U-Net model iterates through 1000 time steps, layer by layer, predicting and eliminating noise components in the feature maps. After final denoising, a pre-trained image decoder remaps the clean latent features back to a high-dimensional pixel space, outputting a high-fidelity predicted X-ray image that accurately reflects the joint degeneration structure after 72 months. Evaluation metrics are automatically calculated on the test set. This embodiment ultimately achieved a 5-class KL rank accuracy of 63.08% over a 72-month span, while controlling the JSW (joint gap width) error to only 1.01%.

[0048] Figure 3 This is an example diagram illustrating the prediction results of the present invention. Figure 3 As shown, from top to bottom, there are five predicted examples of knee joints. The baseline is the X-ray of the input model, and the right side shows 96m images predicted by different methods. The image predicted by our invention (Ours) is closest to the clinical gold standard (ground truth); it is particularly accurate in predicting KOA features such as joint space narrowing and osteophyte formation. The model focuses on areas of interest in the input MRI images (brighter colors indicate higher interest). The model's highly focused areas are concentrated on the patellofemoral joint and infrapatellar fat pad on MRI, which are clinically confirmed precursors to osteoarthritis progression. These findings are consistent with known pathological mechanisms, indicating that the model effectively utilizes clinical data and is interpretable.

[0049] A second aspect of the present invention provides an image prediction system for knee osteoarthritis based on a diffusion model, comprising: The data acquisition module is used to acquire multimodal data of the patient at baseline, including X-ray images, MRI sequences, and clinical features. The model training module is used to adopt a latent diffusion model as the basic prediction network architecture. A multimodal fusion module and cross-attention mask modulation are added to the U-Net structure of the latent diffusion model; AdamW is selected as the optimizer; and the model is trained end-to-end. The image prediction module is used to input the multimodal data into the trained latent diffusion model, perform feature fusion through adaptive gating, and perform inverse diffusion denoising under the constraint of the key point mask of the expanded joint space, and finally decode and output the predicted knee joint X-ray image.

[0050] In one embodiment, the latent diffusion model first utilizes an image encoder with a variational autoencoder (VAE). By compressing real high-dimensional X-ray images into a low-dimensional latent feature space, latent features are obtained. The forward diffusion process gradually adds Gaussian noise to the potential features at time step [number missing]. The noise addition formula is: ,in It is the cumulative product term of the diffusion scalar; the inverse denoising process is performed by a U-Net network parameterized by the model, which is based on the latent representation of the baseline X-ray image. Guided by multimodal conditional information C, noise is predicted and eliminated; the loss function for model training is the mean squared error (MSE). ; After denoising, it is processed by an image decoder. Features are restored to a high-resolution image. .

[0051] In one embodiment, the multimodal fusion module uses a feature extraction network. Extract the first Features of an MRI slice Using a clinical embedding layer Extracting clinical features The fusion features are calculated using an adaptive gating mechanism, and the calculation formula is as follows: ,in, This represents a gating unit used to calculate the dynamic weighting coefficients when clinical features interact with MRI features. Indicates the fused first A multimodal feature vector, after being augmented with positional encoding, is used as a conditional key and value matrix input to the cross-attention layer of the diffusion model.

[0052] In one embodiment, the cross-attention mask modulation includes: locating and extracting key points of the joint space using anatomical landmarks, generating a two-dimensional dilated mask matrix M, and adjusting its size to match the feature map size of the U-Net; in the cross-attention layer, redistributing spatial weights using the mask; and setting... Define the spatial weight matrix for the original cross-attention feature map: ,in, and These are the trainable, independent weight parameters assigned by the system to the inside of the mask (critical lesion area) and the outside of the mask (non-critical background area); the final feature update formula for the l-th layer inside the U-Net is: ,in This indicates element-wise multiplication.

[0053] In one embodiment, the method further includes directly quantifying the predictive accuracy of anatomical degeneration using the Joint Interarticular Distance Error Formula (JSDE), which is used to locate and calculate six key pairs of points between the femur and tibia that best characterize the anatomical structure: ; in, This represents the coordinates of the k-th pair of anatomical key points in the real image x. Represents the generated predicted image The corresponding k-th pair of keypoint coordinates, This formula calculates the Euclidean distance between two points; it also calculates the relative proportional error of the joint gap width between the generated image and the real image.

[0054] While various embodiments of the invention have been shown and described herein, it will be apparent to those skilled in the art that such embodiments are provided by way of example only. Many modifications, alterations, and alternatives will occur to those skilled in the art without departing from the spirit and intent of the invention. It should be understood that various alternatives to the embodiments of the invention described herein may be employed in the practice of the invention. The appended claims are intended to define the scope of protection of the invention and therefore cover modular compositions, equivalents, or alternatives within the scope of these claims.

Claims

1. A method for predicting knee osteoarthritis images based on a diffusion model, characterized in that, include: Acquire multimodal data of the patient at baseline, including X-ray images, MRI sequences, and clinical features; A latent diffusion model is used as the basic prediction network architecture. A multimodal fusion module and cross-attention mask modulation are added to the U-Net structure of the latent diffusion model. AdamW is selected as the optimizer. The model is trained end-to-end. The multimodal data is input into the trained latent diffusion model, feature fusion is completed through adaptive gating, and inverse diffusion denoising is performed under the constraint of the key point mask of the expanded joint space. Finally, the predicted knee joint X-ray image is decoded and output.

2. The image prediction method for knee osteoarthritis according to claim 1, characterized in that: The potential diffusion model: First, the image encoder is utilized using a variational autoencoder (VAE). By compressing real high-dimensional X-ray images into a low-dimensional latent feature space, latent features are obtained. ; The forward diffusion process gradually adds Gaussian noise to the potential features at time step The noise addition formula is: ,in It is the cumulative product term of the diffusion scalar. Represents a normal distribution; This represents the image after adding noise at time step t; This represents the initial, clean image without added noise. The inverse denoising process is performed by a U-Net network that parameterizes the model, which is based on the latent representation of the baseline X-ray image. Guided by multimodal conditional information C, noise is predicted and eliminated; The loss function for model training is the mean squared error (MSE). ； in, Represents the model loss function; This represents the expectation under the conditions x0, xb, C, t, which is the mean of multiple samples in actual training; x0 represents the target image. Represents a baseline image; This represents the actual noise at the current time step t; This represents the noise predicted by the model; express Latent variables obtained via VAE; Represents a baseline image Latent variables obtained via VAE; t represents the denoised time step, and C represents clinical information; After denoising, it is processed by an image decoder. Features are restored to a high-resolution image. .

3. The image prediction method for knee osteoarthritis according to claim 2, characterized in that: The multimodal fusion module: Using feature extraction networks Extract the first Features of an MRI slice ; Using clinical embedding layers Extracting clinical features ; An adaptive gating mechanism is used to calculate the fused features, and the calculation formula is as follows: ， in, This represents a gating unit used to calculate the dynamic weighting coefficients when clinical features interact with MRI features. Indicates the fused first A multimodal feature vector, after adding positional encoding, is used as a condition key and value matrix to be input into the cross-attention layer of the diffusion model; This represents the features extracted from the patient's clinical information by the clinical encoder.

4. The image prediction method for knee osteoarthritis according to claim 3, characterized in that: The cross-attention mask modulation includes: Key points of the joint space are extracted by anatomical landmark localization, a two-dimensional dilated mask matrix M is generated, and its size is adjusted to match the feature map size of U-Net; in the cross-attention layer, spatial weights are redistributed using the mask; set up Define the spatial weight matrix for the original cross-attention feature map: ， in, and These are the trainable, independent weight parameters assigned by the system to the area inside the mask (critical lesion region) and the area outside the mask (non-critical background region); the final feature update formula for the l-th layer inside the U-Net is: ， in, This indicates element-wise multiplication. This represents the (l+1)th layer of the transformer model; The representation extracted by the encoder of VAE; This represents a multilayer sensing network; This represents the mask after weighting by the weights of the w1 joint region and the w2 background region. This represents the attention weight matrix of layer l; This means that the weights are multiplied onto the attention matrix.

5. The image prediction method for knee osteoarthritis according to any one of claims 1-4, characterized in that, It also includes the use of the Joint Space Distance Error Formula (JSDE) to directly quantify the predictive accuracy of anatomical degeneration. This formula is used to locate and calculate six key pairs of points between the femur and tibia that best characterize the anatomical structure. ； in, This represents the coordinates of the k-th pair of anatomical key points in the real image x. Represents the generated predicted image The corresponding k-th pair of keypoint coordinates, This formula calculates the Euclidean distance between two points; it also calculates the relative proportional error of the joint gap width between the generated image and the real image.

6. A knee osteoarthritis image prediction system based on a diffusion model, characterized in that, include: The data acquisition module is used to acquire multimodal data of the patient at baseline, including X-ray images, MRI sequences, and clinical features. The model training module is used to adopt a latent diffusion model as the basic prediction network architecture. A multimodal fusion module and cross-attention mask modulation are added to the U-Net structure of the latent diffusion model; AdamW is selected as the optimizer; and the model is trained end-to-end. The image prediction module is used to input the multimodal data into the trained latent diffusion model, perform feature fusion through adaptive gating, and perform inverse diffusion denoising under the constraint of the key point mask of the expanded joint space, and finally decode and output the predicted knee joint X-ray image.

7. The knee osteoarthritis image prediction system according to claim 6, characterized in that: The potential diffusion model: First, the image encoder is utilized using a variational autoencoder (VAE). By compressing real high-dimensional X-ray images into a low-dimensional latent feature space, latent features are obtained. ; The forward diffusion process gradually adds Gaussian noise to the potential features at time step The noise addition formula is: ,in It is the cumulative product term of the diffusion scalar. Represents a normal distribution; This represents the image after adding noise at time step t; This represents the initial, clean image without added noise. The inverse denoising process is performed by a U-Net network that parameterizes the model, which is based on the latent representation of the baseline X-ray image. Guided by multimodal conditional information C, noise is predicted and eliminated; The loss function for model training is the mean squared error (MSE). ； in, Represents the model loss function; This represents the expectation under the conditions x0, xb, C, t, which is the mean of multiple samples in actual training; x0 represents the target image. Represents a baseline image; This represents the actual noise at the current time step t; This represents the noise predicted by the model; express Latent variables obtained via VAE; Represents a baseline image Latent variables obtained via VAE; t represents the denoised time step, and C represents clinical information; After denoising, it is processed by an image decoder. Features are restored to a high-resolution image. .

8. The knee osteoarthritis image prediction system according to claim 7, characterized in that: The multimodal fusion module: Using feature extraction networks Extract the first Features of an MRI slice ; Using clinical embedding layers Extracting clinical features ; An adaptive gating mechanism is used to calculate the fused features, and the calculation formula is as follows: ， in, This represents a gating unit used to calculate the dynamic weighting coefficients when clinical features interact with MRI features. Indicates the fused first A multimodal feature vector, after adding positional encoding, is used as a condition key and value matrix to be input into the cross-attention layer of the diffusion model; This represents the features extracted from the patient's clinical information by the clinical encoder.

9. The knee osteoarthritis image prediction system according to claim 8, characterized in that: The cross-attention mask modulation includes: Key points of the joint space are extracted by anatomical landmark localization, a two-dimensional dilated mask matrix M is generated, and its size is adjusted to match the feature map size of U-Net; in the cross-attention layer, spatial weights are redistributed using the mask; set up Define the spatial weight matrix for the original cross-attention feature map: ， in, and These are the trainable, independent weight parameters assigned by the system to the area inside the mask (critical lesion region) and the area outside the mask (non-critical background region); the final feature update formula for the l-th layer inside the U-Net is: ， in, This indicates element-wise multiplication. This represents the (l+1)th layer of the transformer model; The representation extracted by the encoder of VAE; This represents a multilayer sensing network; This represents the mask after weighting by the weights of the w1 joint region and the w2 background region. This represents the attention weight matrix of layer l; This means that the weights are multiplied onto the attention matrix.

10. The knee osteoarthritis image prediction system according to any one of claims 6-9, characterized in that, It also includes the use of the Joint Space Distance Error Formula (JSDE) to directly quantify the predictive accuracy of anatomical degeneration. This formula is used to locate and calculate six key pairs of points between the femur and tibia that best characterize the anatomical structure. ； in, This represents the coordinates of the k-th pair of anatomical key points in the real image x. Represents the generated predicted image The corresponding k-th pair of keypoint coordinates, This formula calculates the Euclidean distance between two points; it also calculates the relative proportional error of the joint gap width between the generated image and the real image.

Images