Diabetic foot disease progress recognition method and device based on multi-modal data fusion

Abstract

The invention discloses a diabetic foot disease progress identification method and device based on multi-modal data fusion, and belongs to the technical field of medical artificial intelligence. The method comprises the following steps: positioning and extracting a lesion region of interest (ROI) on an infrared image through a deep learning model by utilizing a visible light image registered with an infrared thermal imaging image; constructing a double-branch network, wherein a visual branch extracts visual features from the extracted ROI by using a convolutional neural network; the text branch converts the structured clinical data into natural language prompts, and uses a CLIP text encoder to extract deep semantic features thereof. The visual features and the clinical semantic features are integrated through a fusion module, and a final prediction result is output through a classifier. According to the method, through image two-stage processing and fusion of local pathological features of the image and global semantic information of clinical data, the ability of distinguishing diabetic patients without complications from diabetic foot patients is remarkably improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of medical artificial intelligence, in particular to a method for medical image analysis using multi-modal information fusion, and more particularly to a diabetic foot disease progression recognition method and device based on multi-modal data fusion. BACKGROUND

[0002] Diabetic foot (DF) is one of the most severe complications of diabetes and is the main cause of non-traumatic lower extremity amputation, posing a great threat to the life and health of patients and their quality of life. According to statistics, there is one amputation surgery due to diabetic foot every 20 seconds worldwide, and the morbidity and mortality rates remain high. Therefore, early detection and diagnosis of diabetic foot disease are of great clinical significance for timely clinical intervention, slowing disease progression, reducing amputation rates, and improving patient outcomes. Clinically, the traditional methods for diagnosing diabetic foot mainly include the 10 gram Semmes-Weinstein monofilament test, percutaneous arterial angiography, etc. However, these methods have their own limitations. For example, the monofilament test is highly subjective, and its results are highly dependent on the experience and operation standardization of clinical doctors; while angiography and other methods are invasive, complex and time-consuming to operate, and carry certain risks, making them unsuitable for large-scale, routine early screening.

[0003] In recent years, infrared thermography (IRT) as a non-invasive, non-radiation, high cost-effective detection method has shown great potential in the medical field. This technology can reflect the subtle temperature changes caused by local blood circulation and metabolic abnormalities by capturing the temperature distribution on the surface of the human body, providing a new perspective for the early pathological state assessment of diabetic foot. With the rapid development of artificial intelligence (AI) technology, combining deep learning with infrared thermography for diabetic foot diagnosis has become a research hotspot. Existing researches mostly use support vector machines (SVM), convolutional neural networks (CNN), etc. to analyze infrared thermography images to distinguish between healthy people and diabetic foot patients. However, these methods generally have the following technical bottlenecks:

[0004] (1) Limitations of diabetic foot classification and diagnosis tasks: existing researches mainly focus on the binary classification of "healthy" and "sick", and fail to effectively address the more critical and complex challenge in clinical practice, i.e., how to distinguish between diabetic patients (DM) without foot complications and diabetic foot patients (DF) with existing lesions. The thermodynamic characteristics of these two groups differ more subtly, requiring higher recognition capabilities from the model.

[0005] (2) The lesion area positioning is missing: most methods directly use the whole foot infrared image as input for classification, which contains a large amount of redundant information of background and non-lesion area. This way not only increases the computational burden, but more importantly, the redundant information may interfere with the model's effective extraction of weak thermal abnormal signals of the key lesion area (such as the early ulcer-prone site), thereby weakening the model's sensitivity and accuracy to early pathological changes.

[0006] (3) Single modal information bottleneck: relying only on infrared thermal imaging, a single modality, it is difficult to comprehensively and accurately assess complex conditions. The occurrence and development of diabetic foot is the result of the joint action of multiple factors, and the patient's age, gender, body mass index (BMI) and other clinical information contain important physiological and pathological background and are the key basis for disease diagnosis. Existing methods fail to effectively fuse these structured clinical data, resulting in limited comprehensive judgment ability of the model to the condition.

[0007] Therefore, there is an urgent need for a new diabetic foot progression recognition method that can accurately locate the lesion area and effectively fuse multi-modal information to overcome the shortcomings of existing technology and improve the accuracy and reliability of early diagnosis. SUMMARY

[0008] The main purpose of the present application is to provide a diabetic foot disease progression recognition method and device based on multi-modal data fusion, aiming to solve the problem of relying only on single modal infrared thermal imaging information in the prior art, lacking accurate positioning of the lesion area, resulting in low accuracy of diabetic foot disease progression recognition.

[0009] To achieve the above purpose, the present application provides the following technical scheme:

[0010] A diabetic foot disease progression recognition method based on multi-modal data fusion, comprising the following steps:

[0011] S1, obtaining multi-modal data of the object to be analyzed, the multi-modal data comprising: an infrared thermal image of the foot , a visible light image of the foot spatially registered with the infrared thermal image , and a descriptive text prompt generated from structured clinical data ;

[0012] S2, lesion area positioning: applying an image positioning model to locate and extract a lesion region of interest image on the infrared thermal image ; ;

[0013] S3, visual feature extraction: processing the lesion region of interest image through a visual feature extraction network ; and parameters thereof to generate a high-dimensional visual feature vector , and the process is as follows:

[0014] ;

[0015] S4, clinical semantic feature extraction: processing the text prompts by a text encoder and parameters thereof to generate a clinical semantic feature vector , and the process is as follows:

[0016] ;

[0017] S5, multi-modal feature fusion: fusing the visual feature vector and the clinical semantic feature vector by a fusion module, and parameters thereof to generate a unified multi-modal fusion feature representation ;

[0018] S6, classification prediction: inputting the multi-modal fusion feature representation into a classifier and parameters thereof to output the final prediction probability of the diabetic foot disease progression state of the object to be analyzed .

[0019] A diabetic foot disease progression recognition device based on multi-modal data fusion, comprising:

[0020] a multi-modal data acquisition module for acquiring multi-modal data of an object to be analyzed, the multi-modal data comprising: an infrared thermal image of the foot , a visible light image of the foot spatially registered with the infrared thermal image , and descriptive text prompts generated from structured clinical data ;

[0021] a lesion area positioning module for applying an image positioning model to locate and extract a lesion region of interest image on the infrared thermal image using image pairs ;

[0022] a visual feature extraction module for processing the lesion region of interest image by a visual feature extraction network and parameters thereof to generate a high-dimensional visual feature vector The process is as follows:

[0023] ;

[0024] The clinical semantic feature extraction module is used to extract features from a text encoder. Process the text prompt and by its parameters Generate clinical semantic feature vectors The process is as follows:

[0025] ;

[0026] A multimodal feature fusion module is used to fuse the visual feature vectors through a fusion module. and clinical semantic feature vector The fusion is performed, and the parameters of the fusion module are used. Generate a unified multimodal fusion feature representation ;

[0027] The classification and prediction module is used to represent the multimodal fusion features. Input to a classifier and by its parameters Output the final predicted probability of the progression status of diabetic foot disease in the object to be analyzed. .

[0028] A computing device includes: at least one processor and a memory storing program instructions; when the program instructions are read and executed by the processor, the computing device performs the method.

[0029] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0030] (1) Improved recognition accuracy: Through an innovative two-stage strategy, the RGB image is first used to guide the precise location of the lesion ROI on the infrared thermogram, effectively eliminating the interference of irrelevant background information, enabling the model to focus more on analyzing the area directly related to pathological changes, and significantly improving the signal-to-noise ratio and effectiveness of feature extraction.

[0031] (2) Enhanced comprehensive judgment ability of the model: This invention creatively transforms structured clinical data into natural language prompts and uses the powerful CLIP text encoder to extract its deep semantics, realizing the deep fusion of visual information and clinical context information. This multimodal fusion method enables the model to comprehensively assess the condition and effectively capture subtle pathological differences that are difficult to detect with single-modal data, especially enhancing the ability to distinguish between DM and DF, two similar states.

[0032] (3) Improved robustness and clinical applicability of the method: This invention provides an end-to-end automated solution with a clear process and strong repeatability from data input to classification result output. This method not only achieved performance far exceeding that of the single-modality baseline model in experiments, but also provides a solid technical foundation for the development of non-invasive, efficient, and low-cost early screening and auxiliary diagnostic tools for diabetic foot, and has broad prospects for clinical application. Attached Figure Description

[0033] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below.

[0034] Figure 1 This is a flowchart of a method for identifying the progression of diabetic foot disease based on multimodal data fusion, according to an embodiment of the present invention.

[0035] Figure 2 This is a schematic diagram of the network architecture according to an embodiment of the present invention;

[0036] Figure 3 This is a schematic diagram of the RGB sample classification results according to an embodiment of the present invention;

[0037] Figure 4 This is a schematic diagram of the classification results of ordinary diabetic patients in an embodiment of the present invention;

[0038] Figure 5 This is a schematic diagram of the classification results of diabetic foot samples according to an embodiment of the present invention. Detailed Implementation

[0039] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

[0040] This invention proposes a novel two-stage multimodal fusion framework called DFP-MMNet (Diabetic Foot Progression MultiModalNetwork) for accurate identification of diabetic foot disease progression. This invention defines the diabetic foot disease progression identification task as a multimodal binary classification problem. Its goal is to distinguish between diabetic patients (DM) without foot ulcers and patients (DF) with diabetic foot ulcers. Given a multimodal input data triplet, including an infrared thermal image... A corresponding registered RGB image and a text prompt generated from structured clinical data. The goal of this invention is to learn a mapping function. This function can predict the correct category label. The category set is .

[0041] The entire method is divided into two stages: the first stage utilizes image pairs Region of interest for lesions The first stage involves the location and extraction of the first element; the second stage is based on the extracted first element. and text prompts The final classification prediction is then performed. Therefore, the mapping function that needs to be optimized can be expressed as: Its parameters By minimizing the cross-entropy loss function To learn.

[0042] Figure 1 This is a flowchart of a method for identifying the progression of diabetic foot disease based on multimodal data fusion, according to an embodiment of the present invention. See below for reference. Figure 1 Describe the execution process of this method. For example... Figure 1 As shown, the method includes:

[0043] S1, acquire multimodal data of the object to be analyzed, the multimodal data including: foot infrared thermal image Visible light image of the foot spatially registered with the infrared thermal imaging image. and descriptive text prompts generated from structured clinical data. ;

[0044] Among them, the multimodal data of the object to be analyzed is obtained. This can include: in a constant temperature In a controlled environment, using a professional infrared thermal imager and a high-resolution RGB camera, infrared thermal images of the soles of each subject's feet were simultaneously captured at a fixed distance (100 cm) and the same angle. and RGB photos To ensure accurate registration of the two images, structured clinical information for each subject was recorded, including at least: age (…). ),gender( ),height( ),weight( Infrared thermal imaging image and RGB photos The dataset was cropped to separate the single-foot views of the left and right feet, forming independent samples. Then, the dataset was strictly divided into training, validation, and test sets in a 7:1:2 ratio.

[0045] S2, Lesion area localization: Applying an image localization model, utilizing image pairs... In the infrared thermal image The image of the region of interest of the lesion was located and extracted. ;

[0046] Since diabetic foot lesions (such as ulcers) typically have more obvious visual features (such as color and texture changes) on RGB images than on infrared images, this invention utilizes this characteristic to guide the precise location of lesion areas on infrared images using RGB images.

[0047] S2 may include: an image localization model employing a deep learning image registration model. The model consists of parameters Definition, in image pairs As input, a binary mask is generated by learning the spatial correspondence between the two. This process can be represented by the following formula:

[0048] ;

[0049] in, These are the network parameters of the registration model. Mask In the diagram, regions with a value of 1 correspond to regions of interest related to lesions, while regions with a value of 0 represent background or non-lesion areas.

[0050] Get the mask Then, it was applied to the original infrared thermal image by element-wise multiplication. This allows for the extraction of the region of interest image of the lesion. :

[0051] ;

[0052] in, This represents element-wise multiplication. The extracted... This will serve as the sole image input for the next stage of visual feature extraction, which greatly reduces the computational load of subsequent processing and eliminates interference from irrelevant information.

[0053] In obtaining Next, the second stage, multimodal feature extraction and fusion classification, begins. This stage comprises two parallel feature extraction branches: a visual feature extraction branch and a clinical semantic feature extraction branch. The goal of the visual feature extraction branch is to extract features from lesion regions of interest images. Highly discriminative visual features are extracted, as described in step S3.

[0054] S3, Visual Feature Extraction: Through a visual feature extraction network Processing the region of interest image of the lesion and by its parameters Generate high-dimensional visual feature vectors The process can be represented as follows:

[0055] ;

[0056] S3 may include: To improve the model's generalization ability and robustness, during training, the input lesion region of interest image is first processed. Perform a series of data augmentation operations, such as random scaling and cropping, and random horizontal flipping; then input the augmented image into a visual feature extraction network. Visual feature extraction network A convolutional neural network (CNN) is used as the visual encoder. In one embodiment, a RegNetY-16GF convolutional neural network model pre-trained on a large image dataset such as ImageNet is selected as the visual feature extraction network. Remove the original classification head of the RegNetY-16GF model and use it only as a feature extractor; visual feature extraction network. Processing lesion region of interest images Then, a high-dimensional visual feature vector is output. (in This process can be represented as:

[0057] ;

[0058] Among them, network parameters Fine-tuning is performed during model training.

[0059] S4, Clinical Semantic Feature Extraction: Through a Text Encoder Process the text prompt and by its parameters Generate clinical semantic feature vectors The process can be represented as follows:

[0060] ;

[0061] The clinical semantic feature extraction branch aims to transform discrete, structured clinical data into rich semantic information that the model can understand and utilize. Step S4 may specifically include:

[0062] S4-1 uses basic clinical data to calculate more informative derivative indicators. For example, based on height... and weight Calculate Body Mass Index (BMI):

[0063] ;

[0064] S4-2, Design a text template function. It is used to fill in the patient's key clinical attributes (age, gender, height, weight, BMI) into a preset sentence structure and generate a fluent and coherent natural language description text. For example, the template could be designed as: "An infrared image of a foot sole from an age-old sex patient who is [height] cm tall and weighs [weight] kg... and their Body Mass Index (BMI) is [bmi]."

[0065] S4-3, the generated natural language description text The input is fed into a pre-trained, powerful text encoder. In this embodiment, the text encoder... This is a CLIP text encoder, a pre-trained language model based on the Transformer architecture. It utilizes the text encoder portion of the CLIP (Contrastive Language Image Pre-Training) model (specifically, a variant of the ViT-B-32-quickgelu). This encoder... Able to transfer text Transform it into a fixed-dimensional clinical feature vector rich in semantics. (in This process can be represented as:

[0066] ;

[0067] To enable the model to better understand medical text relevant to this task, the parameters of the text encoder are... Fine-tuning is also performed during training, but typically a smaller learning rate than that used for the visual encoder is used to preserve its strong pre-trained knowledge.

[0068] S5, Multimodal Feature Fusion: The visual feature vectors are fused together using the fusion module. and clinical semantic feature vector The fusion is performed, and the parameters of the fusion module are used. Generate a unified multimodal fusion feature representation ;

[0069] The goal of this step is to integrate features from different modalities (visual features) and clinical semantic features By effectively combining these elements, a unified and more comprehensive representation of information can be formed.

[0070] This invention employs a simple yet efficient late-fusion strategy, namely feature concatenation. It combines visual feature vectors... and clinical feature vector They are directly connected together to form a longer combined feature vector. : ,in, It is a concatenation function.

[0071] To learn the complex nonlinear relationship between the two modal features, the combined feature vectors are... Input by parameters Defined fusion module In this embodiment, the module is a multilayer perceptron (MLP). This MLP typically includes a batch normalization layer and a layer that reduces the dimensionality to... A linear layer (e.g., 512), a ReLU activation function, and a Dropout layer for regularization. After processing by the fusion module, the final fused feature representation is obtained. :

[0072] ;

[0073] S6, Classification prediction: Represent the multimodal fusion features Input to a classifier and by its parameters Output the final predicted probability of the progression status of diabetic foot disease in the object to be analyzed. .

[0074] The fused feature representation Feed it into a final linear classifier The classifier outputs raw scores (logits) corresponding to each category (DM and DF). (in Then, a Softmax function is used to convert the original scores into predicted probabilities for each category. :

[0075] ;

[0076] Overall parameters of the model End-to-end training is performed using the backpropagation algorithm, with the optimization objective being to minimize the predicted probability. With real labels Cross-entropy loss between them.

[0077] Model training and deployment

[0078] Implementation details: During model training, the input images are uniformly resized to 224×224 pixels. The Adam optimizer is used, and the initial learning rate is set to... The batch size is 32, and the total training duration is 100 epochs. For the CLIP text encoder, which requires fine-tuning, a smaller learning rate is used, such as... The experimental environment was based on the PyTorch 1.13.1 framework and an NVIDIA GeForce RTX 4090 GPU.

[0079] Deployment and Application: After training, the model can be deployed as an auxiliary diagnostic system. After clinicians input the patient's plantar infrared image, RGB image, and basic clinical information, the system can output the probability of the patient having DM or DF in real time, providing objective and quantitative reference for the doctor's diagnostic decision. Table 1 shows the verification of the invention's effectiveness in this specific application environment.

[0080] Table 1. Verification of the Invention's Effects

[0081]

[0082] To verify the effectiveness of the proposed method, extensive experiments were conducted on the dataset. Experimental results show that the proposed DFP-MMNet method achieves an F1 score of 89.3% and an accuracy of 89.2% on the validation set. Compared with single-modal baseline models that only use infrared images (such as RegNetY-16GF, ResNet-50, etc.), the performance improvement exceeds 6%.

[0083] Table 2 Ablation Experiment Results

[0084]

[0085] Ablation experiments further confirmed that the two core innovations of this invention—the two-stage ROI localization strategy and multimodal information fusion—both significantly contributed to the final performance. For example, using only the localized ROIs for classification improved the F1 score from 74.4% to 83.1% based on the baseline. The branch using only clinical text information also achieved an F1 score of 86.7%, demonstrating the high predictive value of clinical data. Finally, the complete model combining both achieved an F1 score of 89.3% (on the validation set), proving the advanced nature and superiority of the technical solution of this invention.

[0086] In summary, this invention provides a complete, innovative, and efficient method for identifying the progression of diabetic foot disease. By combining RGB-guided ROI localization with multimodal (infrared vision + clinical semantic) information fusion, this invention effectively overcomes the limitations of existing technologies, significantly improves the accuracy and robustness of identification, and has important application value in the early, non-invasive, automated screening and assisted diagnosis of diabetic foot.

[0087] The present invention also provides a device for recognizing the progression of diabetic foot disease, comprising:

[0088] The multimodal data acquisition module is used to acquire multimodal data of the object to be analyzed, including: foot infrared thermal imaging image. Visible light image of the foot spatially registered with the infrared thermal imaging image. and descriptive text prompts generated from structured clinical data. ;

[0089] The lesion area localization module is used to apply an image localization model and utilize image pairs. In the infrared thermal image The image of the region of interest of the lesion was located and extracted. ;

[0090] The visual feature extraction module is used to extract visual features through a visual feature extraction network. Processing the region of interest image of the lesion and by its parameters Generate high-dimensional visual feature vectors The process is as follows:

[0091] ;

[0092] The clinical semantic feature extraction module is used to extract features from a text encoder. Process the text prompt and by its parameters Generate clinical semantic feature vectors The process is as follows:

[0093] ;

[0094] A multimodal feature fusion module is used to fuse the visual feature vectors through a fusion module. and clinical semantic feature vector The fusion is performed, and the parameters of the fusion module are used. Generate a unified multimodal fusion feature representation ;

[0095] The classification and prediction module is used to represent the multimodal fusion features. Input to a classifier and by its parameters Output the final predicted probability of the progression status of diabetic foot disease in the object to be analyzed. .

[0096] The present invention also provides a computing device, comprising: at least one processor and a memory storing program instructions; when the program instructions are read and executed by the processor, the computing device performs the method described thereon.

[0097] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for identifying the progression of diabetic foot disease based on multimodal data fusion, characterized in that, Includes the following steps: S1, acquire multimodal data of the object to be analyzed, the multimodal data including: foot infrared thermal image Visible light image of the foot spatially registered with the infrared thermal imaging image. and descriptive text prompts generated from structured clinical data. ; S2, Lesion area localization: Applying an image localization model, utilizing image pairs... In the infrared thermal image The image of the region of interest of the lesion was located and extracted. ; S3, Visual Feature Extraction: Through a visual feature extraction network Processing the region of interest image of the lesion and by its parameters Generate high-dimensional visual feature vectors The process is as follows: ； S4, Clinical Semantic Feature Extraction: Through a Text Encoder Process the text prompt and by its parameters Generate clinical semantic feature vectors The process is as follows: ； S5, Multimodal Feature Fusion: The visual feature vectors are fused together using the fusion module. and clinical semantic feature vector The fusion is performed, and the parameters of the fusion module are used. Generate a unified multimodal fusion feature representation ; S6, Classification prediction: Represent the multimodal fusion features Input to a classifier and by its parameters Output the final predicted probability of the progression status of diabetic foot disease in the object to be analyzed. .

2. The method according to claim 1, characterized in that, S2 includes: The image localization model uses a deep learning registration model. It is composed of parameters Definition, receiving image pairs As input, a binary mask is generated. : ； By using this mask Compared with the original infrared thermal image Perform element-wise multiplication Extracting the region of interest image of the lesion : 。 3. The method according to claim 1, characterized in that, The visual feature extraction network described in step S3 It is a convolutional neural network model RegNetY-16GF pre-trained on a large image dataset.

4. The method according to claim 3, characterized in that, S3 include: First, the input lesion region of interest image is processed. Perform a series of data augmentation operations, including random scaling and cropping, and random horizontal flipping; The enhanced image is then fed into the visual feature extraction network. ; Visual feature extraction network Processing lesion region of interest images Then, a high-dimensional visual feature vector is output. ,in The process is as follows: ； Among them, network parameters Fine-tuning is performed during model training.

5. The method according to claim 1, characterized in that, S4 includes: S4-1 uses basic clinical data to calculate more informative derivative indicators, including: based on height and weight Calculate Body Mass Index (BMI): ； S4-2, Design a text template function. It is used to fill the patient's key clinical attributes, including derived indicators, into a preset sentence structure to generate a fluent and coherent natural language description text. S4-3 inputs the generated natural language description text into a pre-trained text encoder, thereby converting the natural language description text into a fixed-dimensional clinical feature vector with rich semantics.

6. The method according to claim 5, characterized in that, The text encoder described in S4-3 It is a pre-trained language model CLIP text encoder based on the Transformer architecture.

7. The method according to claim 1, characterized in that, S5 include: First, through the splicing operation Visual feature vectors and clinical semantic feature vector Combined into a unified multimodal feature vector : ； Then the multimodal feature vector Input a parameter Defined fusion module A nonlinear transformation is performed to obtain the final fused feature representation. : 。 8. The method according to claim 1, characterized in that, S6 include: The fused feature representation Feed it into a final linear classifier The classifier outputs raw scores corresponding to two categories: diabetic patients without foot complications and diabetic foot patients. ,in Then, a Softmax function is used to convert the raw scores into predicted probabilities for each category. : 。 9. A device for recognizing the progression of diabetic foot disease based on multimodal data fusion, characterized in that, include: The multimodal data acquisition module is used to acquire multimodal data of the object to be analyzed, including: foot infrared thermal imaging image. Visible light image of the foot spatially registered with the infrared thermal imaging image. and descriptive text prompts generated from structured clinical data. ; The lesion area localization module is used to apply an image localization model and utilize image pairs. In the infrared thermal image The image of the region of interest of the lesion was located and extracted. ; The visual feature extraction module is used to extract visual features through a visual feature extraction network. Processing the region of interest image of the lesion and by its parameters Generate high-dimensional visual feature vectors The process is as follows: ； The clinical semantic feature extraction module is used to extract features from a text encoder. Process the text prompt and by its parameters Generate clinical semantic feature vectors The process is as follows: ； A multimodal feature fusion module is used to fuse the visual feature vectors through a fusion module. and clinical semantic feature vector The fusion is performed, and the parameters of the fusion module are used. Generate a unified multimodal fusion feature representation ; The classification and prediction module is used to represent the multimodal fusion features. Input to a classifier and by its parameters Output the final predicted probability of the progression status of diabetic foot disease in the object to be analyzed. .

10. A computing device, characterized in that, include: At least one processor and a memory storing program instructions; When the program instructions are read and executed by the processor, the computing device performs the method as described in any one of claims 1-8.

Images