Pre-diabetes prediction method based on tongue image-retina image bimodal feature fusion

Abstract

The invention discloses a prediabetic prediction method based on tongue image-retina image bimodal feature fusion. The method comprises the steps of constructing a tongue image and retina image pair data set and performing preprocessing; constructing a pre-diabetes prediction model, wherein the pre-diabetes prediction model comprises two multi-scale Mamba modules with the same structure, a bimodal feature fusion module and a classifier; the multi-scale Mama module extracts hierarchical multi-scale semantic features of the tongue image and the retina image of the same patient, the dual-mode feature fusion module is used for splicing and fusion, and the fused features are input into the classifier to output a prediction result of the patient in the early stage of diabetes; training the model through the training set, and monitoring convergence by using the test set to obtain a trained model; and inputting the verification set into the trained model to obtain a pre-diabetes prediction result. The problems that in the prior art, the single medical image feature prediction rate is not high, meanwhile, the convolutional neural network is difficult to capture the long-distance dependency relationship, and the calculation cost based on Transform is high are solved.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence-assisted diagnostic technology, specifically relating to a method for predicting prediabetes based on the fusion of dual-modal features of tongue and retinal images. Background Technology

[0002] Diabetes affects 10% of the global population, is difficult to cure, and can easily lead to over 100 complications affecting the entire body, causing blindness, kidney failure, myocardial infarction, and other serious conditions with very high rates of disability and death. While diabetes itself is incurable, timely intervention in the prediabetic stage can cure or delay its onset. Diabetes symptoms develop slowly, and early symptoms are often difficult to detect, resulting in an "asymptomatic period." This makes it difficult to identify and address the underlying causes. Routine physical examinations often rely solely on blood glucose levels for diagnosis, failing to detect prediabetes. The complex pathogenesis of prediabetes and the varied and often subtle symptoms before the onset of diabetes further complicate the prediction of prediabetes, leading to missed opportunities for optimal treatment. To improve the accuracy and generalization of prediabetes predictions and reduce human error, it is necessary to develop more advanced automated detection technologies to assist or replace manual judgment.

[0003] Currently, hospitals mainly confirm patients' blood sugar levels through multiple blood draws, but multiple blood draws increase the risk of local infection. Existing studies have used tongue appearance or retinopathy symptoms to predict prediabetes, but the pathological correlation of single medical imaging features is weak and the prediction rate is not high. At the same time, methods based on convolutional neural networks (CNN) are difficult to capture long-distance dependencies, resulting in the loss of long-distance feature connectivity, while methods based on Transformers have high computational costs and are difficult to adapt to clinical scenarios. Summary of the Invention

[0004] To overcome the shortcomings of the existing methods and technologies, the present invention aims to provide a method for predicting prediabetes based on the fusion of tongue-retinal image dual-modal features. This method solves the problems of low prediction rate of single medical image features in the existing technology, and the high computational cost of convolutional neural networks in capturing long-distance dependencies.

[0005] This invention is achieved through the following technical solution:

[0006] A method for predicting prediabetes based on bimodal feature fusion of tongue image and retinal image includes the following steps:

[0007] Step 1: Collect tongue and retinal image pairs, construct a tongue and retinal image pair dataset and preprocess it, divide it into training set, test set and validation set, and add annotations to the images in the training set and test set.

[0008] Step 2: Construct a prediabetes prediction model. The prediabetes prediction model includes two structurally identical multi-scale Mamba modules, a bimodal feature fusion module, and a classifier. The two multi-scale Mamba modules are used to extract hierarchical multi-scale semantic features from the tongue image and hierarchical multi-scale semantic features from the retinal image of the same patient, respectively. The bimodal feature fusion module concatenates and fuses the multi-scale semantic features from the tongue image and the retinal image to obtain fused features. The fused features are input into the classifier, which outputs the prediabetes prediction result for the patient.

[0009] Step 3: Train the prediabetes prediction model using the training set and monitor convergence using the test set to obtain the trained prediabetes prediction model.

[0010] Step 4: Input the validation set into the trained prediabetes prediction model to obtain the prediabetes prediction results.

[0011] Furthermore, the dataset in step 1 includes tongue and retinal images of different patients with different categories, symptoms, and conditions; the tongue and retinal image pairs are tongue and retinal images collected from the same patient at the same time period; the image preprocessing specifically involves cropping the tongue and retinal images to the same size and mapping the pixel value range of the images from [0,255] to [-1,1].

[0012] Further, in step 2, the multi-scale Mamba module includes a linear embedding layer, a multi-scale visual state space module, and a block merging module. The linear embedding layer maps the pixel values ​​of the tongue image and retinal lesion image to a high-dimensional feature space, converting the image blocks into a serialized vector representation. The vector representation is input to the encoding part of the multi-scale visual state space module. The four sets of multi-scale visual state space modules capture the contextual features of the image from local to global, extracting global semantic features and fine-grained image features of the tongue image and retinal image. The multi-scale semantic features are respectively input to the three block merging modules to reduce the spatial resolution of the feature map and increase the channel dimension, thereby achieving downsampling between levels and obtaining hierarchical multi-scale semantic features.

[0013] Furthermore, the four sets of multi-scale visual state space modules each include a visual state space module and a multi-scale dilated convolution module. The visual state space module is a stacked structure composed of depthwise separable convolution and a two-dimensional selective scanning module. The depthwise separable convolution is used to extract multi-scale features of the image, and then traverses along four scanning paths to obtain spatial features with global long-range dependencies, while maintaining linear computational complexity. The two-dimensional selective scanning module is used to perform context-enhanced modeling on the spatial features to obtain encoded multi-scale semantic features. The encoded multi-scale semantic features are input into the multi-scale dilated convolution module for multi-branch feature extraction. Through multi-scale dilated convolution, the receptive field of the convolution is gradually expanded to extract global semantic features and fine-grained image features of the tongue image and retina image.

[0014] Further, the dual-modal feature fusion module in step 2 includes two branches. Each branch includes a linear layer, a depthwise separable convolutional layer, a selective state space scanning layer, layer normalization, and channel concatenation connected in sequence. The multi-scale semantic features of the tongue image and the multi-scale semantic features of the retinal image of the same patient are respectively input into the linear layers of the two branches. The linear layer performs dimensional alignment and preliminary transformation on the multi-scale semantic features of the tongue image and the multi-scale semantic features of the retinal image to obtain embedded feature representations. The depthwise separable convolutional layer efficiently extracts local spatial features and reduces computation based on the embedded feature representations to obtain local enhanced features. The selective state space scanning layer models the spatial dependencies of the local enhanced features to obtain context-enhanced features. The layer normalization normalizes the context-enhanced features to obtain stable distribution features. The channel concatenation performs cross-modal concatenation on the stable distribution features to obtain fused features.

[0015] Furthermore, the selective state space scanning layer includes bidirectional state space scanning and feature stitching, which models cross-modal global dependencies while maintaining linear complexity.

[0016] Furthermore, the specific processing procedure of the dual-modal feature fusion module in step 2 is as follows:

[0017] F' M-ton =DWConv(Linear(F M-ton ))

[0018] F' M-ret =DWConv(Linear(F M-ret ))

[0019] S concat =concat(F' M-ton ,F' M-ret )

[0020] S Inverse =Inverse(S concat )

[0021] S' concat =SSM(S concat ),S' Inverse =SSM(S Inverse )

[0022] F ConcatSS =Seperate(S' concat )+Inverse(S' Inverse )

[0023] F' fus =LN(F ConcatSS )+θ1Linear(F M-ton )+θ2Linear(F M-ret )

[0024] F Fus =Linear(F' fus )+F M-ton +F M-ret

[0025] In the formula, F M-ton and F M-ret Let F' represent the multi-scale semantic features of the tongue and retinal images, respectively. `Linear(.)` is the feature linearization operation, `DWConv(.)` is the depthwise separable convolution operation, `concat(.)` is the feature stacking operation, `Seperate(.)` and `Inverse(.)` are the sequence separation and restoration operations, respectively. M-ton and F' M-ret These are the depth-separable convolutional features of the tongue image and the retinal image, respectively. concat and S Inverse S' represents the features after feature stacking and the recovered features, respectively. concat and S' Inverse S represents concat and S Inverse Features after selective space modeling, F ConcatSS F' represents the feature obtained after ConcatSS. fus For F ConcatSS With F M-ton and F M-ret The features are obtained by weighted summation of the features, where θ1 and θ2 are weighting coefficients, and F Fus This refers to the fused features output by the dual-modal feature fusion module.

[0026] Furthermore, the classifier in step 2 is a fully connected structure, which maps the fused features to two neurons and outputs them. The two neurons correspond to the two target categories of the prediction task, namely, patients who are not prediabetic and patients who are prediabetic.

[0027] Furthermore, in step 3, the model is trained using the cross-entropy loss function, specifically as follows:

[0028]

[0029] In the formula, m represents the number of patients, and l i and denoted as the true value and predicted probability of prediabetes for the i-th patient, respectively.

[0030] Furthermore, the Adam optimizer was used for training, with a batch size of 4 and an epoch of 100.

[0031] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0032] This invention addresses the unique characteristics of tongue-retinal images by introducing a multi-scale Mamba module and a bimodal feature fusion model. The multi-scale Mamba module extracts global contextual information and local detail features from both tongue and retinal images, fully leveraging its advantages in long-sequence modeling and multi-scale feature extraction. It simultaneously captures the macroscopic texture features of the tongue and the microscopic vascular details of the retina, achieving efficient cross-modal feature fusion and deep modeling. The bimodal feature fusion module then efficiently fuses the multi-scale features from both images, solving the current challenge of predicting prediabetes. Compared to traditional CNN or Transformer models, this invention demonstrates significant advantages in computational efficiency, feature representation capability, and prediction accuracy, greatly improving the sensitivity and specificity of early screening. It is also practical, easy to operate, and cost-effective, providing an innovative technical solution for large-scale screening and personalized intervention for prediabetes. Furthermore, it opens up new research avenues for bimodal medical data analysis in both traditional Chinese and Western medicine, possessing significant clinical application value and social benefits.

[0033] This invention innovatively integrates the overall metabolic characteristics of tongue diagnosis in Traditional Chinese Medicine with microvascular lesions in the retina from Western medicine, enabling non-invasive and accurate prediction of prediabetes. Attached Figure Description

[0034] Figure 1 This is a structural diagram of the prediabetes prediction model of the present invention.

[0035] Figure 2 This is a structural diagram of the multi-scale visual state space module MSVSS of the present invention.

[0036] Figure 3 This is a structural diagram of the dual-modal feature fusion module of the present invention. Detailed Implementation

[0037] The present invention will be further described in detail below with reference to specific embodiments. These descriptions are for explanation purposes only and are not intended to limit the scope of the invention.

[0038] See Figure 1 A method for predicting prediabetes based on bimodal feature fusion of tongue image and retinal image includes the following steps:

[0039] Step 1: Collect tongue and retinal image pairs, construct a tongue and retinal image pair dataset and preprocess it, divide it into training set, test set and validation set, and add annotations to the images in the training set and test set;

[0040] The dataset includes tongue-retinal images of many patients with different categories, symptoms, and conditions to avoid the problem of weak network generalization ability due to the limited variety of tongue-retinal images. The dataset is also expanded to increase the number of images to avoid overfitting during model training.

[0041] The dataset used in this embodiment of the invention consists of tongue-retinal image pairs collected from the same patients in a hospital clinic. It includes ground truth images of 500 tongue-retinal image pairs from 100 patients. The training set is augmented using image augmentation strategies such as rotation at 90°, 180°, and 270° angles, left-right and up-down mirroring, and shifting by 10 pixels in each of the four directions. This method increases the number of original image pairs tenfold. The images are labeled using the LabelMe tool. Labeled tongue-retinal image pairs are used as training data for model training, while unlabeled image pairs are used as the test set for model testing. Each tongue-retinal image pair consists of one tongue image and one retinal image from the same patient at the same time period.

[0042] The input tongue and retinal images are preprocessed by cropping the input image size to H×W, where H and W are the height and width of the image, respectively, and mapping the pixel value range of the image from [0,255] to [-1,1]. This operation improves the efficiency of the prediction model.

[0043] Step 2: Construct a prediabetes prediction model; the prediabetes prediction model includes two structurally identical multi-scale Mamba modules, a bimodal feature fusion module, and a classifier; each multi-scale Mamba module extracts multi-scale semantic features of the tongue image and multi-scale semantic features of the retinopathy image of the same patient; the bimodal feature fusion module concatenates and fuses the multi-scale semantic features of the tongue image and the multi-scale semantic features of the retinopathy image to obtain fused features; the fused features are input into the classifier, and the prediabetes prediction result of the patient is output;

[0044] Each multi-scale Mamba module consists of a linear embedding layer, four pairs of multi-scale visual state space modules (MSVSS) and three block merging modules. The linear embedding layer maps the pixel values ​​of tongue and retinal lesion images to a high-dimensional feature space, converting image blocks into sequential vector representations. The vector representations are input to the encoding part of the multi-scale visual state space modules. The four sets of multi-scale visual state space modules capture the contextual features of the image from local to global, extracting global semantic features and fine-grained image features of tongue and retinal images. The multi-scale semantic features are respectively input to the three block merging modules to reduce the spatial resolution of the feature map while increasing the channel dimension, achieving downsampling between levels, and obtaining hierarchical multi-scale semantic features.

[0045] Let the preprocessed tongue-retina image pairs be I ton ∈R H×W×3 and I ret ∈R H×W×3 Each image is divided into 16 blocks, and then a linear embedding layer converts the image type into Tensor-type data that the model can accept; I ton and I ret Tensor-type data after the linear embedding layer are input into four MSVSS modules and three block merging modules, respectively. The resolutions of the output feature maps of the four MSVSS modules are H / 4×W / 4×C, H / 8×W / 8×2C, H / 16×W / 16×4C and H / 32×W / 32×8C, respectively, where C is the number of channels.

[0046] Depend on Figure 2As can be seen, MSVSS consists of a State Space Module (VSS) and a Multi-Scale Dilated Convolutional Module (MSDC). MSVSS combines the sequence modeling capabilities of VSS with the multi-scale feature performance of MSDC, processing features at different scales through parallel branches. Each branch uses the VSS module, treating the feature map as a sequence, and captures long-range dependencies using a state space model through spatial and bidirectional scanning. The block merging module reduces the spatial resolution of the feature map while increasing the channel dimension, achieving downsampling between layers. MSDC is a multi-branch feature extraction module, using dilated convolutions with dilation rates of 1, 3, and 5 respectively. The receptive field is gradually expanded by multi-scale dilated convolutions to extract multi-scale features. The MSVSS process can be represented as follows:

[0047] F MSVSS =F MSDC (Linear(F VSS (F in )))

[0048] In the formula, F in and F MSVSS For the input and output characteristics of MSVSS, F MSDC For MSDC operations, the hole ratios are 1, 3, and 5, F VSS (.) represents the output after passing through the VSS module.

[0049] The process of the MSDC module is represented as follows:

[0050] F MSDC =Concat(Conv 1×1 (F0),Conv 3×3 (F0),Conv 5×5 (F0),Conv 7×7 (F0))

[0051] In the formula, Concat(.) is the feature map stacking operation, LN(.) is the layer normalization operation, Linear(.) is the feature linearization operation, F0 is the input of the MSDC module, and F0 = LN(Linear(F VSS )).

[0052] The VSS module is mainly composed of a stacked structure of depthwise separable convolution (DWConv) and two-dimensional selective scanning (2DSS) modules. DWConv extracts multi-scale features of the image and then traverses along four scanning paths to obtain spatial features of global long-range dependencies while maintaining linear computational complexity. The two-dimensional selective scanning module is used to perform context-enhanced modeling on the above spatial features to obtain encoded multi-scale semantic features. The encoded multi-scale semantic features are input into the multi-scale dilated convolution module for multi-branch feature extraction. Through multi-scale dilated convolution, the receptive field of the convolution is gradually expanded to extract global semantic features and fine-grained image features of tongue and retinal images.

[0053] The spatial dimension (H×W) of the two-dimensional feature is decomposed into sequential scans in the horizontal and vertical directions by a 2D selective scan operation (SS2D). One-dimensional selective scans are performed in the horizontal and vertical directions to capture long-distance dependencies. Then, the scan results in the two directions are fused to achieve global context modeling of the two-dimensional image space with linear computational complexity. After another layer of normalization and linear projection, the output is obtained. Global information is captured by squeezing and excitation (SE) operations to optimize feature channel dependencies and obtain multi-scale features.

[0054] Depend on Figure 2 It is known that the VSS module includes a layer normalization module, a linear layer, a depthwise separable convolution DWConv module, SS2D, a layer normalization module, and a squeeze and excitation (SE) module connected in sequence. The obtained features are then connected with a residual and a linear layer to perform channel-weighted integration of bidirectional features, and the output is a visual feature that retains local details and has global context awareness.

[0055] Multi-scale Mamba feature fusion of tongue and retinal images was performed using a dual-modal feature fusion module; the feature fusion process is as follows: Figure 3As shown, after acquiring the multi-scale Mamba features of the tongue-retina images respectively, a dual-modal feature fusion module is used for fusion. The dual-modal feature fusion module takes two inputs: multi-scale semantic features from the tongue and retinal images of the same patient. The output is the fused semantic features, preserving complementary information from both images while suppressing redundant features. The module has two branches, each consisting of a linear layer, a depthwise separable convolution, a selective state space scan layer (ConcatSS), layer normalization, and channel concatenation. The multi-scale semantic features from the tongue and retinal images of the same patient are input to the linear layers of the two branches. These linear layers perform dimensional alignment and preliminary transformation on the tongue and retinal image features to obtain embedded feature representations. The depthwise separable convolution layer efficiently extracts local spatial features based on these embedded feature representations, reducing computation and obtaining local enhanced features. The selective state space scan layer models the spatial dependencies of these local enhanced features to obtain context-enhanced features. Layer normalization normalizes the context-enhanced features to obtain stable distribution features. Channel concatenation performs cross-modal concatenation on these stable distribution features to obtain the fused features.

[0056] The dual-modal feature fusion module achieves bidirectional feature fusion by using ConcatSS to integrate the output features from two multi-scale Mamba modules, preserving the long-range dependencies between different modalities of tongue image and retinal lesion. The linear layer converts the semantic features of tongue image and retinal lesion images into one-dimensional vectors, then concatenates them along the sequence length dimension. The concatenated sequence vectors are then scanned in reverse to obtain two reverse sequence vectors. The original sequence vector and the reverse sequence vectors are then processed to capture long-range dependencies, thereby obtaining a series vector and a reverse series vector. The reverse sequence output is then flipped and added to the processed sequence vector. The summed sequence vector is separated to recover the two outputs, which are then multiplied by two scaling parameters. A linear projection layer is used to reduce the dimensionality of the feature shape to obtain the fused feature vector.

[0057] Let the multi-scale Mamba features of the tongue image and retinal image, after being expanded and concatenated along the sequence length dimension, be represented as F, respectively. M-ton and F M-ret The process of the dual-modal feature fusion module can be represented as follows:

[0058] F' M-ton =DWConv(Linear(F M-ton ))

[0059] F' M-ret =DWConv(Linear(F M-ret))

[0060] S concat =concat(F' M-ton ,F' M-ret )

[0061] S Inverse =Inverse(S concat )

[0062] S' concat =SSM(S concat ),S' Inverse =SSM(S Inverse )

[0063] F ConcatSS =Seperate(S' concat )+Inverse(S' Inverse )

[0064] F' fus =LN(F ConcatSS )+θ1Linear(F M-ton )+θ2Linear(F M-ret )

[0065] F Fus =Linear(F' fus )+F M-ton +F M-ret

[0066] In the formula, DWConv(.) is the depthwise separable convolution operation, concat(.) is the feature stacking operation, Seperate(.) and Inverse(.) are the sequence separation and restoration operations, respectively, and F' M-ton and F' M-ret These are depthwise separable convolutional features, S concat and S Inverse S' represents the features after feature stacking and the recovered features, respectively. concat and S' Inverse S represents concat and S Inverse Features derived from existing selective spatial modeling (SSM), F ConcatSS F' represents the feature obtained after ConcatSS. fus For F ConcatSS With F M-ton and F M-ret The features are obtained by weighted summation of the features, where θ1 and θ2 are weighting coefficients, and F Fus For the fusion feature F' fus With F M-ton and F M-retFeatures after fusion.

[0067] Step 3: Train the prediabetes prediction model using the training set and monitor convergence using the test set to obtain the trained prediabetes prediction model. Train the entire model using the cross-entropy loss function. Prediabetes prediction can be viewed as a binary classification problem, so the two-class cross-entropy loss function in the classifier is used to train the tongue image-retina dual-modal feature fusion model. The loss function is calculated as follows:

[0068]

[0069] In the formula, m represents the number of patients, and l i and denoted as the true value and predicted probability of prediabetes for the i-th patient, respectively.

[0070] Model training details: This model is built on PyTorch, trained using a tongue-retina image pair training set, and convergence monitored using a test set. The Adam optimizer is used to train the entire model. The batch size is set to 4, and the epochs are set to 100.

[0071] Step 4: Input the validation set into the trained prediabetes prediction model to obtain the prediabetes prediction results.

Claims

1. A method for predicting prediabetes based on bimodal feature fusion of tongue image and retinal image, characterized in that, Includes the following steps: Step 1: Collect tongue and retinal image pairs, construct a tongue and retinal image pair dataset and preprocess it, divide it into training set, test set and validation set, and add annotations to the images in the training set and test set. Step 2: Construct a prediabetes prediction model. The prediabetes prediction model includes two structurally identical multi-scale Mamba modules, a bimodal feature fusion module, and a classifier. The two multi-scale Mamba modules are used to extract hierarchical multi-scale semantic features from the tongue image and hierarchical multi-scale semantic features from the retinal image of the same patient, respectively. The bimodal feature fusion module concatenates and fuses the multi-scale semantic features from the tongue image and the retinal image to obtain fused features. The fused features are input into the classifier, which outputs the prediabetes prediction result for the patient. Step 3: Train the prediabetes prediction model using the training set and monitor convergence using the test set to obtain the trained prediabetes prediction model. Step 4: Input the validation set into the trained prediabetes prediction model to obtain the prediabetes prediction results.

2. The method for predicting prediabetes based on bimodal feature fusion of tongue image and retinal image according to claim 1, characterized in that, The dataset in step 1 includes tongue and retinal images of different patients with different categories, symptoms, and conditions; the tongue and retinal image pairs are tongue and retinal images of the same patient collected at the same time period; the image preprocessing specifically involves cropping the tongue and retinal images to the same size and mapping the pixel value range of the images from [0,255] to [-1,1].

3. The method for predicting prediabetes based on bimodal feature fusion of tongue image and retinal image according to claim 1, characterized in that, In step 2, the multi-scale Mamba module includes a linear embedding layer, a multi-scale visual state space module, and a block merging module. The linear embedding layer maps the pixel values ​​of the tongue and retinal lesion images to a high-dimensional feature space, converting image blocks into sequential vector representations. The vector representations are input to the encoding part of the multi-scale visual state space module. The four sets of multi-scale visual state space modules capture the contextual features of the image from local to global, extracting global semantic features and fine-grained image features of the tongue and retinal images. The multi-scale semantic features are respectively input to the three block merging modules to reduce the spatial resolution of the feature map while increasing the channel dimension, achieving downsampling between levels, and obtaining hierarchical multi-scale semantic features.

4. The method for predicting prediabetes based on bimodal feature fusion of tongue image and retinal image according to claim 3, characterized in that, The four sets of multi-scale visual state space modules each include a visual state space module and a multi-scale dilated convolution module. The visual state space module is a stacked structure composed of depthwise separable convolution and a two-dimensional selective scanning module. The depthwise separable convolution is used to extract multi-scale features of the image, and then traverses along four scanning paths to obtain spatial features with global long-range dependencies, while maintaining linear computational complexity. The two-dimensional selective scanning module is used to perform context-enhanced modeling on the spatial features to obtain encoded multi-scale semantic features. The encoded multi-scale semantic features are input into the multi-scale dilated convolution module for multi-branch feature extraction. Through multi-scale dilated convolution, the receptive field of the convolution is gradually expanded to extract global semantic features and fine-grained image features of the tongue image and retina image.

5. The method for predicting prediabetes based on bimodal feature fusion of tongue image and retinal image according to claim 1, characterized in that, In step 2, the dual-modal feature fusion module includes two branches. Each branch includes a linear layer, a depthwise separable convolutional layer, a selective state space scanning layer, layer normalization, and channel concatenation, connected sequentially. Multi-scale semantic features of the tongue image and multi-scale semantic features of the retinal image from the same patient are input into the linear layers of the two branches, respectively. The linear layers perform dimensional alignment and preliminary transformation on the multi-scale semantic features of the tongue image and the retinal image to obtain embedded feature representations. The depthwise separable convolutional layer efficiently extracts local spatial features and reduces computation based on the embedded feature representations to obtain local enhanced features. The selective state space scanning layer models the spatial dependencies of the local enhanced features to obtain context-enhanced features. Layer normalization normalizes the context-enhanced features to obtain stable distribution features. Channel concatenation performs cross-modal concatenation on the stable distribution features to obtain the fused features.

6. The method for predicting prediabetes based on bimodal feature fusion of tongue image and retinal image according to claim 5, characterized in that, The selective state space scanning layer includes bidirectional state space scanning and feature stitching, which models global dependencies across modalities while maintaining linear complexity.

7. The method for predicting prediabetes based on bimodal feature fusion of tongue image and retinal image according to claim 5, characterized in that, The specific processing procedure of the dual-modal feature fusion module in step 2 is as follows: F' M-ton =DWConv(Linear(F M-ton )) F' M-ret =DWConv(Linear(F M-ret )) S concat =concat(F' M-ton ,F' M-ret ) S Inverse =Inverse(S concat ) S' concat =SSM(S concat ),S' Inverse =SSM(S Inverse ) F ConcatSS =Seperate(S' concat )+Inverse(S' Inverse ) F' fus =LN(F ConcatSS )+θ1Linear(F M-ton )+θ2Linear(F M-ret ) F Fus =Linear(F' fus )+F M-ton +F M-ret In the formula, F M-ton and F M-ret Let F' represent the multi-scale semantic features of the tongue and retinal images, respectively. `Linear(.)` is the feature linearization operation, `DWConv(.)` is the depthwise separable convolution operation, `concat(.)` is the feature stacking operation, `Seperate(.)` and `Inverse(.)` are the sequence separation and restoration operations, respectively. M-ton and F' M-ret These are the depth-separable convolutional features of the tongue image and the retinal image, respectively. concat and S Inverse S' represents the features after feature stacking and the recovered features, respectively. concat and S' Inverse S represents concat and S Inverse Features after selective space modeling, F ConcatSS F' represents the feature obtained after ConcatSS. fus For F ConcatSS With F M-ton and F M-ret The features are obtained by weighted summation of the features, where θ1 and θ2 are weighting coefficients, and F Fus This refers to the fused features output by the dual-modal feature fusion module.

8. The method for predicting prediabetes based on bimodal feature fusion of tongue image and retinal image according to claim 1, characterized in that, The classifier in step 2 is a fully connected structure that maps the fused features to two neurons and outputs them. The two neurons correspond to the two target categories of the prediction task, namely, patients who are not prediabetic and patients who are prediabetic.

9. The method for predicting prediabetes based on bimodal feature fusion of tongue image and retinal image according to claim 1, characterized in that, Step 3, which involves training the model using the cross-entropy loss function, specifically involves: In the formula, m represents the number of patients, and l i and denoted as the true value and predicted probability of prediabetes for the i-th patient, respectively.

10. A method for predicting prediabetes based on bimodal feature fusion of tongue-retinal images according to claim 9, characterized in that, Training was performed using the Adam optimizer with a batch size of 4 and an epoch count of 100.

Images