Multi-mode traditional Chinese medicine insomnia differentiation method based on multi-task deep learning

Abstract

The invention discloses a multi-modal traditional Chinese medicine insomnia differentiation method based on multi-task deep learning, which comprises the following steps: S1, collecting tongue images and clinical text symptoms of an insomnia patient, and respectively constructing a data set I and a data set II; s2, based on the data set I and the data set II, constructing and training a multi-mode traditional Chinese medicine insomnia differentiation model; and S3, inputting the standardized tongue picture image and the structured symptom vector of the patient into the multi-mode traditional Chinese medicine insomnia dialectical model to obtain a nine-classification result matched with the traditional Chinese medicine insomnia type. According to the method, tongue body segmentation, cross-modal representation construction, quality perception fusion and multi-task prediction can be automatically completed in actual deployment, and probability distribution of nine types of symptoms and a final conclusion are output on a symptom task.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of computer vision and deep learning, and more particularly, to a multi-modal traditional Chinese medicine insomnia syndrome differentiation method based on multi-task deep learning. BACKGROUND

[0002] Insomnia has become an important problem affecting modern people's health. Epidemiological surveys show that nearly 40% of adults in China have varying degrees of sleep disorders. Insomnia has high prevalence and high recurrence, often accompanied by emotional disorders, cognitive decline and psychosomatic comorbidities, causing significant individual health burden and social and economic costs. Its clinical phenotypes are heterogeneous and diverse, posing challenges to precise stratification and individualized intervention. Traditional Chinese medicine syndrome differentiation shows definite efficacy and good compliance in the treatment of insomnia, emphasizing "treatment based on syndrome differentiation" and "combination of four diagnostic methods". Tongue diagnosis, as an important objective clue, plays a key role in clinical practice. However, the current syndrome differentiation process still relies heavily on the experience and subjective judgment of physicians, lacks unified, reproducible standardized procedures and objective quantitative basis, and has insufficient consistency and repeatability across institutions, limiting the large-scale and standardized promotion of diagnosis and treatment models.

[0003] Tongue image collection is easily affected by factors such as lighting conditions, equipment differences, shooting posture and oral humidity, resulting in exposure deviation, color drift and different resolutions. The regional division and labeling of the tongue body have not been standardized, and there are significant differences in the granularity and coding system of symptom recording. Under the conditions of limited sample size, uneven class distribution and weak labeling, the robustness and generalization ability of model training are challenged. Although existing research attempts to use tongue image for automated auxiliary analysis, it is still mainly based on a single modality, and has not effectively integrated multi-source heterogeneous information such as symptoms. Feature fusion is mostly a "black box" feature, and cross-modal semantic association is difficult to be visualized and quantitatively explained clinically. Most methods only output a single indicator such as "insomnia degree", which cannot meet the requirements of mechanism explainability and traceability for traditional Chinese medicine syndrome differentiation. Related schemes represented by public technologies (such as CN118982709A) have made some progress in improving tongue feature expression (such as hyperspectral partitioning and atlas fusion), but are still limited to tongue image single modality and do not include key symptom information, making it difficult to support syndrome differentiation and other multi-task goals. The model lacks explainability, with output mainly focused on degree evaluation, and lacks feature-to-syndrome causal or contribution path display. At the same time, it does not show the impact of modeling and collection quality differences on feature reliability, limiting the robustness and portability in real clinical environments. With the shortage of experienced traditional Chinese medicine physicians in primary medical institutions and the constraints of computing resources, existing schemes have obvious bottlenecks in clinical landing and popularization at the primary level.

[0004] Based on the above pain points, it is urgent to improve the standardization, automation and consistency of traditional Chinese medicine syndrome differentiation, enhance the robustness and promotion value in multi-center and cross-device scenarios, and provide lightweight and deployable intelligent auxiliary tools for grassroots scenarios. SUMMARY

[0005] It is an object of the present application to solve at least the above problems and / or deficiencies and to provide at least the advantages hereinafter described.

[0006] To achieve these objects and other advantages of the present application, a multi-task deep learning-based multi-modal traditional Chinese medicine insomnia syndrome differentiation method is provided, comprising:

[0007] S1, collecting tongue image and 89 clinical text symptoms of insomnia patients, and constructing data set I and data set II respectively;

[0008] S2, constructing and training a multi-modal traditional Chinese medicine insomnia syndrome differentiation model based on data set I and data set II;

[0009] S3, inputting the standardized tongue image and structured symptom vector of the patient into the multi-modal traditional Chinese medicine insomnia syndrome differentiation model to obtain a nine-classification result suitable for the traditional Chinese medicine insomnia syndrome type;

[0010] The multi-modal traditional Chinese medicine insomnia syndrome differentiation model comprises:

[0011] The tongue image is preprocessed to realize a tongue image segmentation model for end-to-end tongue region automatic segmentation;

[0012] A visual encoder and a symptom encoder are used to extract features from the preprocessed tongue image data and clinical text symptom data;

[0013] The extracted tongue image features and symptom features are aligned, and a multi-task learning model is used to realize syndrome differentiation and cross-modal alignment reconstruction simultaneously through joint analysis.

[0014] Preferably, the visual encoder is configured to extract tongue image features using a ViT model to output a tongue image feature vector f I ;

[0015] f I =W I ·h cls +b I

[0016] In the above formula, f I is the tongue image feature vector output by the ViT model; W I is the weight matrix of the tongue image feature full connection layer; h cls is the CLS label output used by the ViT model to aggregate global tongue image information; and b I is the bias term.

[0017] The symptom encoder structures the clinical text symptom by the following formula to obtain the corresponding symptom feature vector f S :

[0018] f S = W S ·s onehot +b S

[0019] In the above formula, W S is a trainable weight matrix that maps low-dimensional symptoms to a dense semantic space, s onehot is a One-hot encoding vector of the symptom, and b S is a bias term.

[0020] Preferably, the multi-task learning model adopts a two-task alternating training framework, and the two tasks include:

[0021] Task A as the inference backbone, and the task A is used for multi-modal fusion and syndrome classification;

[0022] Task B for improving the inference effect of task A, and the main content of the task B includes: And the region comparison is a common alignment mechanism for the symptom-tongue appearance alignment branch and the tongue appearance-symptom alignment branch; the middle-level perception is only used as a perception consistency constraint in the symptom-tongue appearance alignment branch;

[0023] Wherein, at the inference end, the serial execution is performed in the order of the backbone, and at the training end, the optimization is alternated between task A and task B in batches as the granularity;

[0024] The region comparison refers to the positive and negative comparison of tongue regions and symptom subsets based on tongue segmentation during cross-modal alignment, and the corresponding relationship of tongue regions and symptoms is optimized and measured in combination with the region comparison loss of cosine similarity;

[0025] The middle-level perception refers to the addition of the ViT middle-level feature space perception loss to the pixel reconstruction loss during the reconstruction of symptoms and tongue appearance to ensure that the reconstructed image is consistent with the original image in the semantic feature level;

[0026] Preferably, the region comparison is in the following manner:

[0027] Based on the segmentation of tongue appearance into K regions, and the corresponding of each region with the tongue feature vector f I to obtain the region tongue feature vector f k ;

[0028] Each region is corresponded with the symptom feature vector f SCorresponding is performed to obtain a regional symptom feature vector s j ;

[0029] The fine-grained matching between the tongue image regional features and the symptom features is measured by a regional contrast loss L region , and the cross-modal alignment effect is determined:

[0030]

[0031] In the above formula, P is a positive sample, N is a negative sample, τ is a temperature scaling coefficient for adjusting the steepness of the similarity distribution, σ is a Sigmoid function, sim(f k ,s j ) is a cosine similarity measure of the tongue image regional features and the symptom features in the feature space, k is a regional index, and 1≤k≤K, K represents the total number of regions into which the tongue body is predefinedly divided.

[0032] Preferably, the symptom→tongue image alignment branch contains conditional generation and feature regression, and the processing flow is as follows:

[0033] B01, taking the symptom feature vector f S as a condition, a conditional image generator with a U-ViT architecture outputs a tongue image reconstruction :

[0034]

[0035] In the above formula, is a conditional image generator from symptoms to tongue images;

[0036] The pixel reconstruction loss L pixel of the conditional generation is represented by the following formula:

[0037]

[0038] In the above formula, I is the real original tongue image information;

[0039] The middle-level perception loss L percept is represented by the following formula:

[0040]

[0041] In the above formula, is the middle-level feature representation obtained after inputting the reconstructed tongue image into the ViT model at the lth layer, and ViT l (I) is the feature of the real tongue image I at the lth layer of the ViT, and L mid is a selected layer set of the middle-level feature space;

[0042] B02, the symptom→tongue feature regression is performed by the following formula:

[0043]

[0044] In the above formula, is the predicted tongue feature, FC S2I (·) is a full connection layer for realizing the mapping from symptom to tongue feature;

[0045] Then the symptom→tongue feature regression loss L S2I is characterized by the following formula:

[0046]

[0047] B03, the total loss L of the symptom→tongue alignment branch S→I is characterized by the following formula:

[0048] L S→I = λ SI,pixel · L pixel + λ SI,percept · L percept + λ S2I · L S2I

[0049] In the above formula, λ SI_pixel is the pixel loss weight; λ SI_percept is the perceptual loss weight; L percept is the middle-level perceptual loss; λ S2I is the symptom→tongue feature regression loss weight.

[0050] Preferably, the content of the tongue→symptom alignment branch includes: feature regression and natural language generation, and the processing flow is as follows:

[0051] B11, the tongue→symptom feature regression is performed by the following formula:

[0052]

[0053] wherein, is the predicted symptom feature; FC I2S is a full connection layer for realizing the mapping from tongue to symptom feature, f I is a tongue feature vector;

[0054] Then the tongue→symptom feature regression loss L I2S is characterized by the following formula:

[0055]

[0056] B12, under the condition of tongue feature f I , the text decoder G SThe generated symptom description text sequence y:

[0057] y=G S (f I )

[0058] The text generation loss L text is represented by the following formula:

[0059]

[0060] Wherein, P(·) is the conditional probability distribution of the decoder, t is the time step index, and T is the total length of the sequence; y <t is the sequence of all words before time t;

[0061] B13, tongue appearance→symptom alignment branch total loss L I→S is represented by the following formula:

[0062] L I→S =λ I2S ·L I2S +λ text ·L text

[0063] Wherein, λ I2S is the weight of L I2S ; λ I2S is the weight of L I2S ; λ text is the weight of L text ;

[0064] In the reconstruction process, the cross-modal prediction total loss L cross of bidirectional generation and feature regression between symptoms and tongue appearance is represented by the following formula:

[0065] L cross =LS →I +L I→S

[0066] The overall cross-modal learning loss L cm_total of task B is:

[0067] L cm_total =α·L region +β·L cross

[0068] Wherein, α and β are weight coefficients for balancing regional-level alignment and global-level cross-modal prediction.

[0069] Preferably, the multi-modal fusion of task A refers to: adopting a dynamic fusion mechanism of quality gating network to fuse the cross-modal aligned tongue appearance features and symptom features, and obtaining the final fused feature vector f through the following formula:

[0070] f = g o f I + (1-g) o f S

[0071] In the above formula, o represents element-wise multiplication, f I is a tongue feature vector, f S is a symptom feature vector; o represents element-wise multiplication, f I is a tongue feature vector, f S is a symptom feature vector, 1-g is the inverse weight of the gating vector, g is the quality-aware gating vector, and g = σ(Wg[f I ‖f S ‖q I ‖q S ]+b g );

[0072] In the above formula, W g is a gating network weight, σ is a Sigmoid function, ‖ is concatenation, b g is a bias term, q S is a symptom quality score, q I is a tongue quality score, and the tongue quality score q I is obtained by the following formula:

[0073]

[0074] In the above formula, α1, α2, α3 are weight coefficients, and α1+α2+α3 = 1, exp is an exposure value; exp0 is a standard exposure value, Δexp max is a maximum exposure deviation, blur is a sharpness, and κ is a segmentation confidence;

[0075] The symptom quality score q S is obtained by the following formula:

[0076] q S = β1·comp+β2·consist+β3·freshness

[0077] In the above formula, β1, β2, β3 are weight coefficients, and β1+β2+β3 = 1, comp is a symptom completion degree, consist is a symptom description consistency, and freshness is a time freshness.

[0078] Preferably, the syndrome classification of task A is based on the fused feature vector f, and the syndrome classification probability vector p c is calculated, and the classification loss value L cls is used to measure the difference between the predicted result and the true label to determine the classification effect of the model, and at the same time complete the inference output of the multi-task;

[0079] wherein the syndrome type classification probability vector p c is characterized by the following formula:

[0080] p c = Softmax(W cls ·f+b cls )

[0081] In the above formula, W cls is a classification weight matrix that maps the fused features to each syndrome type category, and b cls is a classification bias vector that adjusts the basic tendency of each category.

[0082] Preferably, when the quality gating network is used to fuse the tongue feature and the symptom feature, a quality consistency loss L task is introduced on the basis of the original syndrome classification and cross-modal reconstruction task loss set L quality , which constrains the unreasonable situation of assigning high weight to low quality modalities, and then the total fusion loss L fusion is characterized by the following formula:

[0083] L fusion = L quality + λ fusion L task

[0084] In the above formula, λ fusion is a weighting coefficient, and the quality consistency loss L quality is characterized by the following formula:

[0085] L quality = mean(g)·(1-q I ) 2 + mean(1-g)·(1-q S ) 2

[0086] In the above formula, mean(g) is the average weight assigned to the tongue feature f I output by the gating network in the fusion process, (1-q I ) 2 represents the penalty for low quality tongue, and mean(1-g) is the average value output by the gating network in the fusion process, which represents the overall degree of trust of the model to the symptom modality; (1-q S ) 2 represents the penalty for low symptom quality.

[0087] The total loss in the training stage is:

[0088] L train = λ cls ·Lcls +λ cm ·L cm_total +λ q ·L quality

[0089] wherein, λ cls , λ cm , λ q are weights corresponding to each loss respectively, L cls is a classification loss, and L cls = CrossEntropy(p c , c true ), c true is a real syndrome category label.

[0090] Preferably, the inference end performs inference prediction and explainability output, wherein the inference prediction part refers to: based on the maximum a posteriori principle, the maximum value in the classification probability vector p c is output as the syndrome conclusion;

[0091] The inference prediction and explainability output include: when selecting to view the symptom-tongue image alignment branch, generating a corresponding tongue image reconstruction graph conditioned on symptom embedding and outputting pixel and middle layer perception consistency scores for intuitive comparison by doctors;

[0092] When selecting to view the tongue image-symptom alignment branch, output the probability distribution generated by symptom text and the contribution degree of key phrases;

[0093] The results of the symptom-tongue image alignment branch and the tongue image-symptom alignment branch cooperate with SHAP to perform post-hoc explanation on the syndrome head, to quantify the marginal contribution of tongue image feature dimensions and symptom keywords to target syndrome prediction, and then provide Top-M feature ranking, combine regional level similarity heat maps, output the correlation strength and visual superimposed graph of tongue area and symptoms, and assist clinical interpretation and review.

[0094] The present application at least includes the following beneficial effects:

[0095] Firstly, the present application can automatically complete tongue segmentation, cross-modal representation construction, quality perception fusion and multi-task prediction in actual deployment, and output the probability distribution and final conclusion of nine types of syndromes on the syndrome task.

[0096] Secondly, the present application also provides clinically-oriented explainability results to assist clinical interpretation and review, facilitating the application of doctors' implementability;

[0097] Thirdly, the process of the present application can be realized in a conventional GPU / CPU environment, and has good scalability and reviewable characteristics for primary application.

[0098] Additional advantages, objects, and features of the application will be apparent from the following description, and upon practice of the application in its various embodiments. BRIEF DESCRIPTION OF DRAWINGS

[0099] Figure 1 A flowchart of a multi-task deep learning-based multi-modal traditional Chinese medicine insomnia syndrome differentiation method of the application;

[0100] Figure 2 A segmentation result graph after preprocessing of a tongue image of the application;

[0101] Figure 3 A schematic diagram of cross-modal alignment and reconstruction of the application. DETAILED DESCRIPTION

[0102] The application will be further described in detail below with reference to the accompanying drawings, so that those skilled in the art can implement the application according to the description and the drawings.

[0103] The application proposes a multi-task deep learning-based multi-modal traditional Chinese medicine insomnia syndrome differentiation method based on computer vision and deep learning methods, which mainly multi-modal fusion of tongue diagnosis image features and traditional Chinese medicine clinical symptom data is realized, and the objective evaluation of syndrome differentiation is realized. It can integrate tongue and structured symptoms under a unified framework, use quality-aware dynamic fusion mechanism to distinguishably absorb reliable information, and depict the fine-grained correlation between tongue area and symptoms through regional-level alignment; in the aspect of explainability, provide visualization and quantitative evidence for physician decision-making, such as the contribution of tongue feature and symptom keyword to syndrome type prediction, the correlation strength between tongue area and symptoms, etc., to enhance the understandability and reviewability of the results, such as Figure 1 as shown, specifically comprising the following steps:

[0104] 1. Collect patient tongue image and clinical symptom data through standardized equipment, in the tongue image analysis process, first use LabelMe to label tongue contour to generate binary mask and extract color tongue area, then train end-to-end deep learning segmentation network based on these labeled data, finally obtain professional tongue image segmentation model which can accurately segment tongue area.

[0105] Specifically, as shown in Figure 2 in the input and preprocessing stage, a standardized imaging device with a unified light source is used to obtain patient front natural tongue image, and 89 traditional Chinese medicine symptoms are recorded synchronously and structured.

[0106] There are 89 clinical text symptoms, as shown in Table 1:

[0107] Table 1 Clinical symptom coding table

[0108]

[0109]

[0110]

[0111] The image is pixel-wise labeled for tongue contour using LabelMe, generating a binary mask; accordingly, a segmentation network (U-Net / DeepLabV3+ / nnUNet) is trained to obtain a tongue image segmentation model, which automatically segments the tongue and crops the color tongue region during inference, uniformly scales to 224x224 and completes intensity normalization. The tongue image segmentation model is automatically called as a pre-module during the whole inference process to ensure stable input of the tongue region.

[0112] Further, the tongue image segmentation model uses a deep convolutional neural network to realize end-to-end automatic segmentation of the tongue region, supporting the selection and comparison of three mainstream network architectures:

[0113] (1) U-Net architecture: based on the symmetric structure design of encoder-decoder, the encoder gradually down-samples through consecutive convolution and pooling operations to extract multi-level semantic features; the decoder up-samples through transposed convolution to restore the spatial resolution; the key is the skip connection, which directly transmits the feature maps of each layer of the encoder to the corresponding layer of the decoder, effectively preserving the detailed information and avoiding information loss in the down-sampling process.

[0114] (2) DeepLabV3+ architecture: combining Atrous Convolution and Spatial Pyramid Pooling (ASPP) modules, it captures multi-scale context information through parallel Atrous Convolution with different dilation rates, expanding the receptive field without increasing the number of parameters; using an encoder-decoder structure, the encoder extracts rich semantic features, and the decoder gradually restores spatial details.

[0115] (3) nnUNet architecture: an adaptive optimization version based on U-Net, which automatically adjusts network depth, kernel size, batch size, and other hyperparameters according to dataset characteristics through an automated network configuration strategy; integrates advanced data preprocessing procedures, including adaptive resampling, intensity normalization, and data augmentation strategies; optimized specifically for medical image segmentation tasks, improving training efficiency and model generalization while maintaining high accuracy.

[0116] The tongue image segmentation model is trained using a combined loss function for optimization:

[0117] L seg =λ dice ·L dice +λ ce ·Lce +λ boundary ·L boundary

[0118] where L dice is Dice loss, which is used to specifically deal with the class imbalance problem commonly seen in medical images, and optimize the segmentation accuracy by calculating the overlap degree of the prediction and the true label; L ce is cross-entropy loss, which is used to provide pixel-level classification supervision; L boundary is boundary loss, which is used to focus on improving the accuracy of the segmentation edge and to fine-tune the constraint of the tongue contour; λ dice , λ ce , λ boundary are the corresponding weight coefficients, and the optimal ratio is determined through experiments.

[0119] 2. Based on the pre-trained ViT (Vision-Transformer) model, the tongue features are extracted, and the 89 symptom data are structured and coded;

[0120] In terms of tongue feature extraction, the ViT model (12-layer Transformer, hidden layer 768 dimensions) is used, the input tongue image is standardized and cropped to 224x224 pixels, and then block-embedded into a 768-dimensional vector. The global feature corresponding to the CLS label is extracted, and the dimension is reduced to 256 dimensions through the fully connected layer.

[0121] f I =W I ·h cls +b I

[0122] The above formula is the whole representation of the linear mapping of the CLS label feature h cls of the ViT to the "tongue feature vector" f I , f I is the output 256-dimensional tongue feature vector; W I is the tongue feature fully connected layer weight matrix, with dimensions 256x768, which is a linear algebra operation of "matrix multiplied by vector"; h cls is the CLS label output of the ViT, with dimensions 768, which aggregates global tongue information; b I is the bias term, with dimensions 256.

[0123] The symptom feature vector is a 256-dimensional semantic feature obtained by structured coding of the 89 TCM symptoms, and its calculation process is as follows:

[0124] f S =W S ·s onehot +b S

[0125] where f S is the output 256-dimensional symptom feature vector; W S is the trainable weight matrix with dimension 256x89, mapping 89-dimensional symptoms to a 256-dimensional dense semantic space; s onehot is the one-hot encoded vector of symptoms with dimension 89; b S is the bias term with dimension 256.

[0126] 3. Jointly analyze tongue image features and symptoms based on a multi-task learning model, and simultaneously realize syndrome differentiation and cross-modal reconstruction;

[0127] The application adopts a two-task alternating training framework: task A is multi-modal fusion and syndrome differentiation classification (reasoning backbone and deployment task), and task B is cross-modal reconstruction (region comparison is symptom Tongue image universal alignment mechanism; middle-level perception is only used for the symptom to tongue image (S to I) branch). The training end alternately optimizes between task A and task B in batches or rounds: in the task A step, the fusion and classification related loss is optimized, and in the task B step, the region comparison and reconstruction related loss is optimized; the reasoning end is only serially executed in the order of the backbone, and the explainable results (probability and SHAP region heat map) are output in the deployment stage; each loss is controlled by the weight of the hyperparameter λ and stable convergence and performance balance are realized through alternating update.

[0128] (1) Multi-modal fusion stage

[0129] The syndrome differentiation task first needs to fuse the aligned data of the two modalities, and this step adopts quality-aware dynamic fusion (Quality-aware Modal Fusion, QMF). In order to avoid the "wrong dominance" caused by low-quality input (such as tongue image blur, unstable exposure or incomplete symptom record) to joint analysis, this link introduces the collection quality score in the joint representation of tongue image features and symptom features, dynamically adjusts the contribution of each modality, and realizes quality-aware adaptive fusion.

[0130] The tongue image feature vector is denoted as f I (from the ViT encoder); the symptom feature vector is denoted as f S (from the structured encoding / embedder).

[0131] The tongue image quality score q I ∈[0,1], is obtained by weighting sub-indices such as segmentation confidence κ, clarity (blur), exposure stability (exp), etc., so that:

[0132]

[0133] wherein, a1, a2, a3 are weight coefficients, and a1+a2+a3=1, exp is an exposure value, referring to the normalized mean value of brightness within the tongue mask (preferably calculated in CIELab space Lightness channel and linearly normalized to [0, 1]); exp0 is a standard exposure value, referring to the exposure reference of the reference sample collected under standard illumination and white balance conditions (taking the mean value of the reference sample set); Δexp max is the maximum exposure deviation, referring to the allowable maximum difference relative to exp0, used for normalizing the exposure deviation |exp-exp0| (preferably set as three times the standard deviation of the calibration sample exposure value, and the ratio is clipped to [0, 1]); blur is the sharpness, referring to the normalized quantity based on the sharpness index within the tongue mask; κ is the segmentation confidence;

[0134] symptom quality score q S ∈[0, 1], obtained by weighting the sub-indices of symptom filling completeness (comp), symptom description consistency (consist), time freshness (freshness), etc., then:

[0135] q S = β1·comp+ β2·consist+ β3·freshness,

[0136] wherein, β1, β2, β3 are weight coefficients, and β1+β2+β3=1, comp is the symptom filling completeness, which is the symptom filling completeness (value [0, 1], calculated by the weighted record proportion of the mandatory item set S), i.e.:

[0137]

[0138] wherein, M is the mandatory item set; w i is the weight of the i-th mandatory item; m i is an indication quantity of whether the i-th mandatory item is filled in;

[0139] consist is the symptom description consistency, referring to the compliance score of the pre-defined constraint set in the one-hot encoded symptom vector (preferably, only one "1" is allowed in the mutually exclusive single-choice group, and the cross-domain logic is not contradictory, for example, "no dream" and "dream much" cannot be selected together, if multiple "1"s appear in a group or "normal" and "abnormal" are selected together, it is counted as a conflict);

[0140]

[0141] wherein, G is the constraint group set; u r is the weight of the constraint group r; c rThe satisfaction indicator of the constraint group r is recorded as 1 if the constraint group satisfies the constraint group r, and 0 otherwise; γ is a conflict penalty coefficient for controlling the deduction strength of the conflict term; C conflict The count of cross-domain contradictory pairs (such as "no dream" and "dream more" and selected);C max is a normalization constant (upper limit or upper confidence interval of the conflict count, used to normalize the conflict term to [0, 1]).

[0142] freshness time freshness, which refers to the timeliness score of the symptom record from the current visit time, that is:

[0143]

[0144] Where Δt is the time interval from the visit (preferably in "days", Δt ≥ 0); τ fresh is a timeliness scale parameter (τ fresh > 0, used to control the decay rate). freshness ∈ (0, 1], when the record time is closer to the visit (Δt → 0), it tends to 1, and it decays exponentially with the increase of the time interval.

[0145] The dynamic fusion mechanism adopts quality gating network fusion: let

[0146] g = σ (W g [f I ‖f S ‖q I ‖q S ]+b g )

[0147] Where W g is the gating network weight, σ is the Sigmoid function, ‖ is splicing, b g is the bias term.

[0148] Then the fusion feature is:

[0149] f = g ⊙ f I +(1-g)⊙f S

[0150] Where f represents the final fused feature vector; g is the quality-aware gating vector, which is dynamically generated by the gating network according to the tongue feature, symptom feature and their respective quality scores, and its value range is [0, 1], which represents the trust degree of the model to the tongue feature; (1-g) is the inverse weight of the gating vector, which represents the trust degree of the model to the symptom feature; ⊙ represents element-wise multiplication.

[0151] To suppress the unreasonable situation that low-quality modalities are given high weights, a quality consistency constraint is introduced:

[0152] L quality= mean(g) · (1 - q I ) 2 + mean(1 - g) · (1 - q S ) 2

[0153] where L quality is the quality consistency loss, used to constrain the consistency between the fusion weight and the quality score of each modality, to prevent low-quality modalities from being given too high a weight; mean(g) is the average value of the output of the gating network, representing the average weight given to the tongue image feature f I in the fusion process; (1 - q I ) 2 represents the penalty for low-quality tongue images, and the value of this term increases as the quality of the tongue image becomes worse; mean(1 - g) is the inverse average value of the output of the gating network, representing the degree of trust of the model in the symptom modality as a whole in the fusion process; (1 - q S ) 2 represents the penalty for low-quality symptoms, and the value of this term increases as the quality of the symptom becomes worse.

[0154] (2) Cross-modal alignment and reconstruction phase

[0155] This phase contains two directions:

[0156] Symptom→tongue image (S→I): fine-grained alignment is performed using region contrast, and ViT mid-level perception loss is added outside the pixel reconstruction loss, i.e., mid-level perception is only used as a perception consistency constraint in the symptom→tongue image branch, to ensure that the generated tongue image is semantically consistent with the original Figure 1 ;

[0157] Tongue image→symptom (I→S): fine-grained alignment is performed using region contrast, without introducing mid-level perception constraints, and contains two types of objectives, feature regression and natural language generation, for inferring symptom embeddings and text descriptions from tongue image features; and only the subset of symptoms related to tongue image visibility is supervised and displayed (mask M_vis), to avoid overstepping into non-visual fields, while also serving to reasonably complete missing / low-quality modalities.

[0158] Task B0, i.e., region alignment, is performed by segmenting the tongue image to obtain K semantic regions R k , and then extracting the corresponding region features f k (such as ), the middle layer features from the first layer of ViT are taken and aggregated according to the region mask, in order to ensure the implementability and consistency of the "correspondence between each region and the tongue feature vector", the tongue body is divided into K predefined regions in this paper, each region is cut by the tongue binary mask along the axis and the edge according to the fixed proportion and geometric rules, ensuring that the region masks are mutually exclusive and collectively cover the tongue body; then each region mask is aligned with the middle layer feature grid of the visual encoder: in the visual ViT scenario, the mask is projected to the patch (image block) grid and the patch features belonging to the region are normalized and weighted aggregated to obtain the middle layer representation of the region; then the dimension is unified through the linear mapping layer to obtain the regional tongue feature vector, and the vector is preferably normalized to facilitate the similarity measurement and contrast training with the regional symptom vector; in this process, each region mask needs to be strictly aligned in the same coordinate system and not overlap with each other. Through the above spatial indexing and weighted aggregation process based on the region mask, each regional tongue feature vector only represents the image semantics of the region, and can be aligned, compared and fused with the regional symptom feature vector at the regional level;

[0159] Through the symptom embedding method, each region and the symptom feature vector f S are corresponded to obtain the regional symptom feature vector s j , and the symptom subset corresponding to each region is recorded as S k , and in order to ensure the implementability and consistency of the "correspondence between each region and the symptom feature vector", the symptom encoder is used to structure the embedding of the clinical text symptom to obtain the symptom vector set; the regional condition is used as a query (given by the image region feature), and attention weighted aggregation is performed on the above symptom embedding set to obtain the symptom middle layer representation of the region; and the dimension is unified through the linear mapping layer to obtain the regional symptom feature vector. The attention weight is determined by normalized similarity calculation, and symptom quality weight can be introduced to suppress the influence of incomplete or low consistency symptoms on the aggregation result. The above correspondence process is realized in the unified coordinate and dimension, so that each regional symptom feature vector only represents the symptom semantics related to the region, and is aligned, compared and fused with the regional tongue feature vector in the same feature space.

[0160] Therefore, the cosine similarity is:

[0161]

[0162] Where, sim(f k ,s j ) is used to measure the similarity of the tongue region feature and the symptom feature in the feature space; f k represents the feature vector of the kth tongue region, which is extracted by the visual encoder for the kth segmented region of the tongue image, reflecting the semantic information of the region; s jThe feature vector representing the jth symptom, extracted by the symptom structured encoder for the jth symptom data, reflecting the semantic information of the symptom.

[0163] Let P = (f k ,s j ) be a positive sample pair, and N = (f j ,s k ) be a negative sample pair. The cosine contrast of the log-likelihood formula is adopted:

[0164]

[0165] Where P is the positive sample set, indicating the "tongue region-symptom" pair with clinical relevance to a specific tongue region, N is the negative sample pair set, indicating the "tongue region-symptom" pair with no clinical relevance to a specific tongue region, τ is the temperature scaling coefficient used to adjust the steepness of the similarity distribution, σ is the Sigmoid function, sim(f k ,s j ) is the cosine similarity of the tongue region features and the symptom features in the feature space, k is the region index, K represents the total number of regions predefined for the tongue, and 1 ≤ k ≤ K. The first term in the formula represents the log-likelihood term for the "relevant region-symptom" sample pair, and minimizing this term is equivalent to increasing the similarity of the positive sample pair; the second term in the formula represents the log-likelihood term for the "irrelevant region-symptom" sample pair, which is equivalent to log(1-σ(τ·sim)) by introducing a negative sign in the input of σ, and minimizing this term is equivalent to lowering the similarity of the negative sample pair. The total region contrast loss L region ranges from 0 to +∞, i.e., non-negative and unbounded above; it can approach 0 in ideal alignment (high positive sample similarity and low negative sample similarity).

[0166] Based on the above region contrast loss L region , the method constructs a complete cross-modal learning framework, as shown in Figure 3 . This framework includes two levels of learning objectives: (1) fine-grained alignment at the region level, which realizes accurate matching of tongue region and symptom features through L region ; (2) cross-modal prediction at the global level, which realizes mutual conversion between symptoms and tongue images through the bidirectional generation task. The two levels complement each other and jointly optimize cross-modal representation learning.

[0167] For On the basis of region-level alignment, cross-modal prediction is divided into two parts according to direction: symptom→tongue image (S→I) and tongue image→symptom (I→S). Among them, the middle-level perception and pixel domain reconstruction are only used in the S→I branch, which is used for perception consistency and pixel-level generation constraints; the I→S branch does not contain middle-level perception and pixel reconstruction, and adopts feature regression and natural language generation. The region contrast is a fine-grained alignment mechanism common to the two branches, which specifically includes:

[0168] Task B1, i.e. symptom→tongue image (S→I) alignment branch, mainly includes conditional generation (pixel level) and feature regression (feature level), and the specific processing flow is as follows:

[0169] 1) Conditional generation (pixel level): take the symptom feature f S as the condition, and output the tongue image reconstruction (adopting U-ViT architecture, containing multi-scale upsampling and skip connection) by the conditional image generator:

[0170]

[0171] Where f S is the symptom feature vector obtained by the symptom encoder; G I adopts a conditional decoding structure (U-ViT decoder), and f S is injected into the decoding path through attention mechanism to ensure the semantic modulation of text content to tongue image reconstruction.

[0172] Pixel reconstruction loss L pixel :

[0173]

[0174] In the above formula, I is the real original tongue image information, is the predicted tongue image of the conditional image generator G I , and G I (·) is the conditional image generator from symptom to tongue image.

[0175] 2) Middle-level perception (perception consistency constraint):

[0176]

[0177] In the above formula, is the middle-level feature representation obtained after inputting the reconstructed tongue image into the ViT model at the lth layer; ViT l (l) is the middle-level feature extraction function of the ViT model at the lth layer, and ViT l (·) is the middle-level feature extraction function of the lth layer of the ViT model, and L midis the selected mid-level feature space layer set. The mid-level perceptual constraint and the aforementioned region contrast loss form a complementary multi-level alignment mechanism: region contrast realizes accurate matching of local regions in spatial dimension, and mid-level perception guarantees semantic consistency of generated images in feature dimension.

[0178] 3) Feature regression (S→I, feature level):

[0179]

[0180] wherein, is the predicted tongue feature; FC S2I is the fully connected layer, symptom to tongue feature mapping;

[0181] L S2I Symptom to tongue feature regression loss:

[0182]

[0183] 4) S→I symptom to tongue branch total loss:

[0184] L S→I = λ SI_pixel · L pixel + λ SI_percept · L percept + λ S2I · L S2I

[0185] wherein, L S→I is the symptom to tongue branch total loss, λ SI_pixel is the pixel loss weight; L pixel is the pixel reconstruction loss; λ SI_percept is the mid-level perception loss weight; L percept is the perception loss; λ S2I is the S2I feature regression loss weight; L S2I is the symptom to tongue feature regression loss.

[0186] Task B2, i.e., tongue→symptom (I→S) alignment branch, mainly includes: feature regression (feature level) and natural language generation (text level), and the specific processing flow is as follows:

[0187] 1) Feature regression (I→S, feature level):

[0188]

[0189] wherein, is the predicted symptom feature; FC I2S is the fully connected layer, tongue to symptom feature mapping; f I is the tongue feature vector;

[0190]

[0191] Among them, L I2S The regression loss is due to the loss of tongue appearance to symptom features. For predicted symptom characteristics; f S For symptom feature vectors;

[0192] 2) Natural Language Generation (Text Level): Based on tongue image features f I As a condition, a symptom description sequence is generated by the text decoder:

[0193] y = G S (f I )

[0194] Where y is the symptom description text sequence, G S For text generator;

[0195] Text generation loss (sequence cross-entropy):

[0196]

[0197] Where y is the symptom description text sequence, P(·) is the conditional probability distribution of the decoder, t is the time step index, and T is the total sequence length; <t G represents the sequence of words preceding time t; text generator G S The Transformer decoding structure is pre-trained under conditions, and teacher forcing can be used during the training phase.

[0198] 3) Total loss L from tongue appearance to symptom branch in I→S I→S :

[0199] L I→S =λ I2S ·L I2S +λ text ·L text

[0200] Where, λ I2S L represents the loss weights for I2S feature regression; I2S For tongue appearance to symptom feature regression loss; λ text Generate loss weights for the text; L text Generate loss for the text;

[0201] Total cross-modal prediction loss for tasks B1 and B2:

[0202] L cross =L S→I +L I→S

[0203] The overall cross-modal learning loss of task B is:

[0204] L cm_total = a L region + b L cross

[0205] wherein a and b are weight coefficients for balancing the importance of regional-level alignment and global-level cross-modal prediction. region L cross achieves fine-grained matching between tongue region and symptom characteristics, L c achieves bidirectional generation and feature regression between symptoms and tongue.

[0206] (3) Inference prediction and explainable output stage

[0207] The inference prediction is a syndrome classification based on multi-task learning at the inference end, and the corresponding prediction result is given. The specific operation mode is to perform insomnia syndrome classification based on the fusion feature f:

[0208] p c = Softmax(W cls f + b cls )

[0209] wherein p c represents a syndrome classification probability vector, representing the probability distribution of the input sample belonging to each insomnia syndrome category; W cls is a classification weight matrix with a dimension of 9x256, used to map the fusion feature to each syndrome category; b cls is a classification bias vector with a dimension of 9, used to adjust the basic tendency of each category.

[0210] Classification loss:

[0211] L cls = CrossEntropy(p c , c true )

[0212] wherein L cls represents the classification loss value, used to measure the difference between the prediction result and the true label; CrossEntropy is the cross-entropy loss function; c true is the true syndrome category label. In the training stage, L cls is minimized to improve the model's confidence in the true category.

[0213] (4) Output the syndrome conclusion using the SHAP explainability analysis method:

[0214] 1) For the syndrome model: calculate the contribution of tongue feature and symptom keyword to syndrome prediction.

[0215]

[0216] In the above formula, φ i is the SHAP value of the i-th sample feature f; N is the number of samples; is the SHAP contribution value of the i-th feature in the n-th sample.

[0217] The output is the contribution ranking of each tongue feature and symptom keyword, which satisfies the additivity constraint:

[0218]

[0219] where d represents the number of all features; f(x) represents the predicted output value of the model for the current input sample x; E[f(x)] represents the expectation operator, which is the average prediction value of the model on all training samples; f(x)-E[f(x)] represents the difference between the current sample prediction value and the baseline value, i.e. the deviation of the current sample relative to the average level.

[0220] 2) Cross-modal reconstruction: quantifying the association strength between tongue region and symptoms.

[0221]

[0222] In the above formula, A i,j is the association strength between tongue modal feature i and symptom modal feature j; i is the tongue modal feature index; j is the symptom modal feature index; N c is the number of samples of the same syndrome type;

[0223] is the SHAP value of tongue feature i in the n-th sample; is the SHAP value of symptom feature i in the n-th sample.

[0224] Embodiment

[0225] The embodiment provides an intelligent tongue and symptom differentiation method based on multi-task learning, which realizes an end-to-end process from data acquisition, feature construction, cross-modal alignment and quality perception fusion, to cross-modal reconstruction and explainable output in a single deployable framework. The system input is a standardized tongue image and a structured symptom vector of a patient, and the output includes insomnia syndrome type nine classification results (heart and spleen deficiency, dampness and heat obstruction, dampness and heat down, heart and liver blood deficiency, liver stagnation and spleen deficiency, heart and kidney disharmony, yin deficiency with dampness, yin deficiency with fire, liver and kidney deficiency, as a nine-class multi-class classification task of a single classification head) and its probability distribution, visual explanation and tongue region-symptom association analysis. The specific processing flow is:

[0226] 1、In the input and preprocessing stage, a standardized imaging device with a uniform light source is used to obtain the patient's natural tongue-out image, and 89 TCM symptom records are collected and structured synchronously. The image is pixel-by-pixel labeled using LabelMe to generate a binary mask; accordingly, a segmentation network (U-Net / DeepLabV3+ / nnUNet) is trained to automatically segment the tongue and crop the color tongue region during inference, and the region is uniformly scaled to 224x224 and intensity normalized. The segmentation model is automatically called as a pre-module during the whole inference process to ensure stable input of the tongue region.

[0227] 2、Based on the multi-task learning model, the tongue features and symptoms are jointly analyzed to realize the syndrome differentiation and cross-modal reconstruction simultaneously.

[0228] In the feature representation construction stage, the tongue modality uses a pre-trained ViT model as a visual encoder to embed the 224x224 image in blocks, and the global representation h cls corresponding to the CLS label is obtained by encoding, and a 256-dimensional tongue feature vector f I is obtained by two-layer full connection projection and normalization; at the same time, the regional features of several representative layers are selected for subsequent middle-layer perception and regional-level alignment.

[0229] The 89-dimensional One-hot symptom vector s onehot is mapped to a 256-dimensional semantic embedding f S through a trainable embedding matrix, and is equipped with a nonlinear transformation and layer normalization to improve the stability of the representation. To realize cross-modal semantic alignment, a contrastive learning constraint is introduced: (f I ,f S ) of the same case as a positive sample pair, and negative sample pairs are constructed within and across batches, and through the InfoNCE loss of cosine similarity, the similarity of the corresponding pairs is improved, and the similarity of the non-corresponding pairs is reduced, so that the tongue and symptom representations are aligned in the public semantic space, and the modality consistency before fusion is enhanced.

[0230] In the multi-modal fusion stage, a quality-aware dynamic fusion mechanism is used to model the reliability of the tongue and symptoms. Specifically, the quality scores q I and q S are estimated through a lightweight quality assessment subnetwork for f I and f S , respectively, and a 256-dimensional gating vector g is generated through a gating network. The fusion feature is defined as f. To prevent the dominance of low-quality modalities, a quality consistency loss L quality is set to make the gating vector statistically consistent with (q I , q S ); at the same time, entropy regularization is introduced to suppress extreme "single modality", leaving a cross-modal complementary space on difficult samples.

[0231] In the cross-modal reconstruction and region-level alignment stage, to enhance the fine-grained semantic association between tongue region and symptoms, the system further divides the segmentation results into several tongue regions R corresponding to anatomy and clinical function. k (e.g., tip of the tongue, sides of the tongue, middle of the tongue, root of the tongue, etc.), and based on clinical knowledge, map the symptom set into several subject subsets S. k For each tongue region, extract the mid-layer region features f of ViT. k The similarity is calculated with the corresponding symptom subset embeddings, and the region contrast loss L is optimized. region This ensures that related tongue regions and symptoms are close together in the representation space. To improve cross-modal consistency and interpretability, a generative reconstruction branch from symptoms to tongue image is constructed, integrating pixel-level reconstruction loss and ViT mid-level perceptual loss: the former ensures the reconstruction image... Fidelity is maintained at the color and shape levels, the latter constraining the mid-level representation to retain structural and textural semantics consistent with the real tongue image I. This reconstruction branch is periodically invoked during the training phase, in turn promoting the alignment of symptoms with tongue image representations.

[0232] In the reasoning and interpretability output stage, the fused feature f is used as input: the symptom classification head outputs the probability distribution of nine symptom types. (Corresponding to "deficiency of both heart and spleen, damp-heat obstruction in the middle jiao, damp-heat descending to the lower jiao, deficiency of heart and liver blood, liver stagnation and spleen deficiency, disharmony between heart and kidney, yin deficiency with dampness, yin deficiency with fire excess, and deficiency of liver and kidney"), the inference end gives the final diagnostic conclusion based on the maximum a posteriori principle; the training end uses cross-entropy loss to alleviate class imbalance. The system sequentially calls the segmentation model to complete automatic tongue cropping, and calls the visual encoder and symptom encoder to obtain (f I ,f S The probability of nine symptom types is obtained by f after passing through a quality subnet and a gating network, and then called upon the classification head to output the probability. Simultaneously, SHAP is used to perform posterior interpretation of the symptom head, quantifying the marginal contribution φ of tongue image feature dimensions and symptom keywords to the prediction of the target symptom type. i It also provides a Top-M ranking of important features; combined with a region-level similarity heatmap, the system outputs the correlation strength A between tongue region and symptoms. i,j Overlaying with visualizations aids in clinical interpretation and verification.

[0233] Example 2

[0234] For the convenience of those skilled in the art to understand, a literal end-to-end example is given: the input is a tongue image I = "ID_DATE.jpg" of an adult patient, collected under standard light source, and after automatic segmentation, a tongue body color area of 224x224 is obtained; at the same time, input into the structured symptom vector S (89 items), assume that the items such as "difficulty falling asleep, easy to wake up, stomach distension, fatigue, loss of appetite, palpitation, and lack of color" are 1, and the rest are 0 or low intensity value. Next, first call the segmentation model to complete tongue extraction and cropping; then call the ViT encoder to obtain h cls and obtain Call the symptom embedding layer to obtain The inference stage outputs q I = 0.82, q S = 0.74 by the quality evaluation subnetwork; the gating network generates g according to this, and the fusion feature f is obtained by weighting the two modalities; further call the syndrome classification head to obtain the probability distribution of nine categories, for example: heart and spleen deficiency 0.63, liver stagnation and spleen deficiency 0.17, heart and liver blood deficiency 0.08, and the rest 0.12, according to the maximum a posteriori principle, output the syndrome conclusion as "heart and spleen deficiency". At the same time, the system calls the explanation module to return the SHAP contribution bar chart and local explanation text: the symptoms side "stomach distension, fatigue, loss of appetite, and palpitation" contribute positively to "heart and spleen deficiency", and the middle layer feature channel of the tongue side "tongue body is slightly pale, edge tooth marks, and thin white fur" has a large positive contribution to the conclusion; the region-symptom heat map shows that the tongue and tongue edge regions are strongly related to the "fatigue / loss of appetite" theme subset. If the user chooses to view the cross-modal reconstruction, the system calls the reconstruction branch to generate the corresponding tongue image reconstruction with pixel and middle layer perception consistency score, for the doctor to visually compare.

[0235] This embodiment can automatically complete tongue segmentation, quality perception fusion, multi-task prediction, and cross-modal representation construction in actual deployment, and provide clinically interpretable results; output the probability distribution of nine types of syndromes and the final conclusion on the syndrome type task, thereby improving the standardization, automation, and clinical usability of TCM syndrome differentiation. This process can be implemented in a regular GPU / CPU environment, and has good scalability and reviewable characteristics for primary applications.

[0236] In addition, in order to avoid misuse and medical risks, the main use (missing / low-quality modal completion) of cross-modal prediction in this embodiment is as follows: when the tongue image is missing or of low quality (such as strong reflection, severe blur, segmentation failure), the system can generate a tongue schematic / reconstruction according to the structured record on the symptom side to participate in the reconstruction and alignment task, helping to maintain the consistency of cross-modal representation; when the symptom record is incomplete or of low quality, the system only reasonably completes the items that have a visual correlation with the tongue image, without touching the demographic information and items that have no direct correlation with the tongue surface.

[0237] The generated or completed results are only used as auxiliary signals in training to characterize alignment and improve robustness, and do not replace real acquisition data. When encountering a missing or low-quality input of a certain modality at the inference end, the system will perform an explicit prompt and, if necessary, participate in fusion and interpretation with a generated approximate modality, but the interface clearly labels it as a "generated / completed result" and does not replace real acquisition;

[0238] Items without reliable visual basis are uniformly labeled as "not based on tongue appearance judgment / need clinical input" to avoid misguiding; demographic information (such as gender, age, and disease duration) is always kept as clinical metadata; and explainability is provided synchronously (for example, the corresponding relationship between tongue area and symptoms is prompted, and it is explained which tongue clues support which judgment), which facilitates physician review and teaching use.

[0239] When the tongue appearance is missing but the symptoms are recorded completely, the system can generate a tongue body sketch for reconstruction and alignment tasks in training, and at the inference end, the consistency between the text explanation and the sketch is prompted, and the generated image is not used as a diagnostic basis; when the symptoms are not complete but the tongue appearance is clear, the system only completes the items related to the tongue appearance (such as "tooth marks, moss, cracks, etc.), and clearly prompts "clinical input" for gender, age, and disease duration.

[0240] The above methods ensure that the cross-modal prediction is used for "robust completion of missing / low-quality modalities", while maintaining strict boundaries and labeling specifications.

[0241] The above scheme is only a description of a preferred example, but is not limited thereto. In the implementation of the present application, appropriate replacement and / or modification can be made according to the needs of the user.

[0242] Although the embodiments of the present application have been disclosed as above, it is not limited to the applications listed in the specification and embodiments. It can be fully applied to various fields suitable for the present application. Additional modifications can be easily realized by those skilled in the art. Therefore, the present application is not limited to specific details and the figures shown and described herein, without departing from the general concept defined by the claims and their equivalent scope.

Claims

1. A multimodal TCM insomnia diagnosis method based on multi-task deep learning, characterized in that, include: S1. Collect tongue images and 89 clinical text symptoms from insomnia patients to construct dataset I and dataset II respectively; S2. Based on dataset I and dataset II, construct and train a multimodal TCM insomnia diagnosis model; S3. Input the patient's standardized tongue image and structured symptom vector into the multimodal TCM insomnia diagnosis model to obtain a nine-category result that matches the TCM insomnia syndrome type; The multimodal TCM insomnia diagnosis model includes: Preprocessing is performed on tongue images to achieve an end-to-end automatic segmentation model for tongue regions; A visual encoder and a symptom encoder for feature extraction from preprocessed tongue image data and clinical text symptom data; The extracted tongue features and symptom features are aligned, and a multi-task learning model for syndrome differentiation and cross-modal alignment reconstruction is realized simultaneously through joint analysis.

2. The multimodal TCM insomnia diagnosis method based on multi-task deep learning as described in claim 1, characterized in that, The visual encoder is configured to use the ViT model for tongue image feature extraction, so as to output the tongue image feature vector f by the following formula. I ; f I =W I ·h cls +b I In the above formula, f I W is the tongue image feature vector output by the ViT model. I h is the weight matrix of the fully connected layer for tongue image features; cls The CLS-labeled output used by the ViT model to aggregate global tongue image information; b I For bias terms; The symptom encoder performs structured encoding of clinical text symptoms using the following formula to obtain the corresponding symptom feature vector f. S : f S =W S ·s onehot +b S In the above formula, W S To map low-dimensional symptoms to a trainable weight matrix in a dense semantic space, s onehot b is the one-hot encoded vector of the symptom. S This is a bias term.

3. The multimodal TCM insomnia diagnosis method based on multi-task deep learning as described in claim 1, characterized in that, The multi-task learning model employs a two-task alternating training framework, wherein the two tasks include: Task A serves as the main reasoning framework, and Task A is used for multimodal fusion and dialectical classification; Task B is used to improve the reasoning performance of Task A. The main contents of Task B include: Furthermore, regional comparison serves as a common alignment mechanism for both the symptom-tongue image alignment branch and the tongue image-symptom alignment branch; mid-level perception is only used as a perceptual consistency constraint in the symptom-tongue image alignment branch. In this process, the inference process is executed sequentially according to the main steps, while the training process optimizes alternately between task A and task B in batches. The region comparison refers to the positive and negative comparison between the tongue region and the symptom subset obtained from tongue image segmentation during cross-modal alignment, and the region comparison loss based on cosine similarity is used to optimize and measure the correspondence between the tongue region and the symptom. The mid-level perception refers to the superposition of ViT mid-level feature space perception loss on top of pixel reconstruction loss during symptom and tongue image reconstruction to ensure that the reconstructed image remains consistent with the original image at the semantic feature level.

4. The multimodal TCM insomnia diagnosis method based on multi-task deep learning as described in claim 3, characterized in that, The method for comparing the regions is as follows: Based on segmenting the tongue image into K regions and associating each region with the tongue image feature vector f I To obtain the regional tongue image feature vector f, a corresponding mapping is performed. k ; By embedding symptoms, each region is associated with the symptom feature vector f. S Perform the mapping to obtain the regional symptom feature vector s j ; Then, the area contrast loss L in the following formula is used. region This allows for the measurement of fine-grained matching between tongue image region features and symptom features, thereby determining the cross-modal alignment effect. In the above formula, P represents positive samples, N represents negative samples, τ is a temperature scaling factor used to adjust the steepness of the similarity distribution, σ is the Sigmoid function, and sim(f k ,s j To measure the cosine similarity between tongue region features and symptom features in the feature space, k is the region index, and 1≤k≤K, where K represents the total number of predefined regions of the tongue body.

5. The multimodal TCM insomnia diagnosis method based on multi-task deep learning as described in claim 4, characterized in that, The content of the symptom → tongue image alignment branch is: conditional generation and feature regression, and the processing flow is as follows: B01, using symptom feature vector f S As a condition, the conditional image generator using the U-ViT architecture outputs a reconstructed tongue image using the following formula. In the above formula, G I A conditional image generator for symptom-to-tongue image mapping; Then the pixel reconstruction loss L during conditional generation pixel Characterized by the following formula: In the above formula, I represents the actual original tongue image information; Mid-layer perception loss L percept Characterized by the following formula: In the above formula, To reconstruct the tongue image After being input into the ViT model, the mid-level feature representation obtained in the l-th layer is ViT. l (I) represents the features of the real tongue image I in the 1st layer of ViT, L mid This refers to the set of layers selected in the mid-level feature space; B02. Regression of symptom-tongue appearance characteristics using the following formula: In the above formula, For predicted tongue features, FC S2I (·) is a fully connected layer for mapping symptoms to tongue appearance features; The feature regression loss from symptoms to tongue appearance is L S2I Characterized by the following formula: B03, Symptoms → Total loss of tongue image alignment branches L S→I Characterized by the following formula: L S→I =λ SI,pixel ·L pixel +λ SI,percept ·L percept +λ S2I ·L S2I In the above formula, λ SI_pixel λ represents the pixel loss weights. SI_percept For perceived loss weights; L percept For mid-level perception loss; λ S2I The feature regression weights from symptoms to tongue appearance are used.

6. The multimodal TCM insomnia diagnosis method based on multi-task deep learning as described in claim 3, characterized in that, The tongue image → symptom alignment branch includes: feature regression and natural language generation, and the processing flow is as follows: B11. Perform feature regression analysis of tongue appearance and symptoms using the following formula: in, For predicted symptom characteristics; FC I2S To achieve a fully connected layer that maps tongue appearance to symptom features, f I This represents the tongue image feature vector; Then the characteristic regression loss L for tongue appearance → symptoms I2S Characterized by the following formula: B12, based on tongue characteristics f I As a condition, via text decoder G S Generate a symptom description text sequence y: y=G S (f I ) Then the text generation loss L text Characterized by the following formula: Where P(·) is the conditional probability distribution of the decoder, t is the time step index, and T is the total sequence length; y <t The sequence of all words before time t; B13. Tongue appearance → Total loss of branches aligned with symptoms (L) I→S Characterized by the following formula: L I→S =λ I2S ·L I2S +λ text ·L text Where, λ I2S For L I2S The weights; λ text For L text The weights; Achieving bidirectional generation and feature regression between symptoms and tongue images, and cross-modal prediction of total loss L cross Characterized by the following formula: L cross =L S→I +L I→S Then the overall cross-modal learning loss L for task B cm_total : L cm_total =α·L region +β·L cross Where α and β are weighting coefficients used to balance regional alignment and global cross-modal prediction.

7. The multimodal TCM insomnia diagnosis method based on multi-task deep learning as described in claim 6, characterized in that, Task A's multimodal fusion refers to: using a dynamic fusion mechanism with a quality-gated network to fuse cross-modal aligned tongue image features and symptom features, and obtaining the final fused feature vector f using the following formula: f=g⊙f I +(1-g)⊙f S In the above formula, ⊙ represents element-wise multiplication, and f I Let f be the tongue image feature vector. S For symptom feature vectors; ⊙ represents element-wise multiplication, f I Let f be the tongue image feature vector. S Let be the symptom feature vector, 1-g be the inverse weight of the gating vector, g be the quality-perceived gating vector, and g = σ(W). g [f I ||f S ‖q I ‖q S ]+b g ); In the above formula, W g Here, σ represents the weights of the gated network, ‖ represents the sigmoid function, and b represents the concatenation. g For the bias term, q S For symptom quality scoring, q I Tongue image quality score, and tongue image quality score q I We obtain it from the following formula: In the above formula, α1, α2, and α3 are weighting coefficients, and α1 + α2 + α3 = 1; exp is the exposure value; exp0 is the standard exposure value; and Δexp max The maximum exposure deviation is represented by blur, and the segmentation confidence level is κ. Symptom quality score q S We obtain it from the following formula: q S =β1·comp+β2·consist+β3·freshness In the above formula, β1, β2, and β3 are weighting coefficients, and β1 + β2 + β3 = 1. comp represents the completeness of symptom completion, consistent represents the consistency of symptom description, and freshness represents the time freshness.

8. The multimodal TCM insomnia diagnosis method based on multi-task deep learning as described in claim 7, characterized in that, The dialectical classification of Task A is based on the fused feature vector f, and the probability vector p of the syndrome type classification is calculated. c And through the classification loss value L cls The model's classification performance is determined by measuring the difference between the predicted results and the true labels, while also completing the inference output for multiple tasks. Wherein, the probability vector for the type classification is p c It is characterized by the following formula: p c =Softmax(W cls ·f+b cls ) In the above formula, W cls To map the fused features to the classification weight matrix of each symptom category, b cls This is used to adjust the classification bias vectors for the basic tendencies of each category.

9. The multimodal TCM insomnia diagnosis method based on multi-task deep learning as described in claim 8, characterized in that, When using a quality-gated network to fuse tongue appearance features and symptom features, the loss set L of the original syndrome differentiation and cross-modal reconstruction task is... task Based on this, introduce quality consistency loss L quality If constraints are imposed on the unreasonable situation where low-quality modes are given high weights, the total fusion loss L will be reduced. fusion It is characterized by the following formula: L fusion =L quality +λ fusion L task In the above formula, λ fusion As a trade-off factor, the quality consistency loss L quality It is characterized by the following formula: L quality =mean(g)·(1-q I ) 2 +mean(1-g)·(1-q S ) 2 In the above formula, mean(g) represents the tongue image features f output by the gating network during the fusion process. I The assigned average weight, (1-q) I ) 2 The penalty for low-quality tongue images is represented by mean(1-g), which is the inverse average of the gating network output during the fusion process, representing the overall confidence level of the model in the symptom modality; (1-q) S ) 2 This indicates a punishment for low symptom quality; The total loss during the training phase is: L train =λ cls ·L cls +λ cm ·L cm_total +λ q ·L quality Where, λ cls , λ cm , λ q These are the weights corresponding to each loss, L. cls For classification loss, and L cls =CrossEntropy(p c ,c true ), c true This is the actual symptom category label.

10. The multimodal TCM insomnia diagnosis method based on multi-task deep learning as described in claim 3, characterized in that, At the inference end, inference prediction and interpretable output are performed. The inference prediction part refers to: based on the maximum a posteriori principle, the classification probability vector p is... c The maximum value in the equation is output as the dialectical conclusion. The inference prediction and interpretability output include: when selecting the view symptoms → tongue image alignment branch, generating the corresponding tongue image reconstruction map based on the symptom embedding, and outputting the pixel and mid-level perception consistency score for doctors to make intuitive comparisons. When selecting to view the tongue image → symptom alignment branch, output the probability distribution of symptom text generation and the contribution of key phrases; The results of the symptom-tongue alignment branch and the tongue-symptom alignment branch are combined with SHAP symptom head for posterior interpretation to quantify the marginal contribution of tongue feature dimensions and symptom keywords to the prediction of the target symptom type, thereby providing Top-M feature ranking. Combined with regional similarity heatmaps, the correlation strength and visualization overlay of tongue region and symptom are output to assist clinical interpretation and verification.

Images