Traditional Chinese medicine syndrome differentiation multi-modal data processing method and system based on inference chain distillation

Abstract

The application discloses a traditional Chinese medicine syndrome differentiation multi-modal data processing method and system based on an inference chain distillation, the method encodes four diagnosis modal data into modal projection vectors, inputs a shared syndrome differentiation feature vector obtained by weighted summation according to a modal weight vector into a teacher model and a student model containing multiple inference stages, a decoder of each stage generates a hidden state of the stage from a hidden state of a previous stage and the vector, and calculates transition probability distributions of three actions of advancing, backtracking and terminating according to cross attention weights of the modal projection vectors; when a termination probability is maximum, the inference is stopped and a mode to be supplemented is prompted; when a backtracking probability is greater than an advancing probability and a backtracking number does not reach an upper limit, the modal weight is updated according to the cross attention weight, the feature vector is recalculated, and the previous stage is re-executed, so that misleading evidence of a low-quality mode is not amplified along the chain; the inference chain distillation is completed by aligning transition probability distributions of the two models, and structured syndrome differentiation reference information is output from the student model.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology in traditional Chinese medicine, specifically to a method and system for processing multimodal data of TCM syndrome differentiation based on inference chain distillation. Background Technology

[0002] Traditional Chinese medicine's (TCM) diagnostic and treatment principles are the core theoretical system of TCM, and the diagnostic thinking of renowned veteran TCM doctors is crucial for the inheritance of TCM. In the traditional TCM diagnosis and treatment process, doctors obtain patient information through the four diagnostic methods of observation, auscultation and olfaction, inquiry, and palpation. This information is then processed through multiple stages of reasoning, including pathogenesis analysis, syndrome differentiation, treatment method establishment, and prescription organization, to form a complete diagnostic plan. This process highly relies on the doctor's clinical experience and diagnostic thinking ability, exhibiting significant personalized and tacit knowledge characteristics.

[0003] With the development of artificial intelligence technology, existing technologies have explored various aspects of intelligent TCM syndrome differentiation. For example, patent CN118800406A discloses a multimodal TCM clinical identification and treatment data sharing service system and method. This system acquires multimodal TCM clinical data, including medical record text, tongue images, facial images, pulse images, and audio recordings. After data cleaning, standardization, and feature extraction, feature-level fusion is used to merge the feature vectors of different modalities into a unified feature vector. The multimodal fusion model then outputs pathological identification results, generating diagnostic and treatment decision information, including TCM prescriptions. Patent CN109102882A discloses a TCM syndrome differentiation and treatment system. This system encodes each symptom name into a quantitative code containing information on etiology, location, nature, and progression. After superposition to obtain the main symptom quantitative code, syndrome differentiation is performed to obtain the syndrome type. Based on the syndrome type, the treatment method and corresponding TCM prescription are retrieved from the database, realizing sequential reasoning from symptom to syndrome to treatment to prescription to medicine. Patent CN111611377A discloses a training method and apparatus for a multilayer neural network language model based on knowledge distillation. It not only aligns the probability distribution of the final output layer between the teacher model and the student model, but also aligns the representations of the embedding layer and each intermediate hidden layer. It solves the problem of inconsistent hidden dimensions between the teacher and the student through a learnable linear transformation matrix, and the loss of each layer is weighted and summed according to the weight coefficient as the distillation target.

[0004] However, the aforementioned existing technologies suffer from the following technical bottlenecks. First, the multimodal fusion weights are disconnected from the dialectical conclusions. In schemes like CN118800406A, each modal feature is assigned a fixed weight during the fusion process, preventing the fusion layer from knowing the downstream dialectical conclusions. This leads to the continuous amplification of misleading evidence from low-quality or weakly correlated modalities. Second, if the preceding stage of a multi-stage dialectical chain is flawed, the subsequent stages can only perpetuate the error. In schemes like CN109102882A, the stages are cascaded deterministically in a unidirectional manner, lacking a path back to the upstream for re-analysis. Errors extracted in the preceding stage propagate along the chain, and backtracking is mathematically meaningless. Third, knowledge distillation cannot transfer reasoning path decisions. In schemes like CN111611377A, transferable knowledge only includes intermediate layer representations and final output probabilities. Since deterministic reasoning chains lack action space, the teacher's path decisions have no carrier for distillation, and the student model can only learn the conclusions, not the paths. In addition, large and complex models have a large number of parameters and high inference latency, which cannot meet the real-time deployment needs of primary healthcare institutions; traditional knowledge distillation only focuses on the alignment of the final output, which makes it impossible for student models to reproduce the dialectical path, and multi-stage one-time distillation is also prone to catastrophic forgetting. Summary of the Invention

[0005] This invention proposes a method and system for processing multimodal data of TCM syndrome differentiation based on inference chain distillation, which solves the problem in the prior art that multimodal fusion, multi-stage reasoning and knowledge distillation are separated from each other, and that the downstream syndrome differentiation conclusions cannot have a feedback effect on the upstream modal fusion and stage reasoning.

[0006] To address the aforementioned technical problems, this invention provides a method for processing multimodal data of TCM syndrome differentiation based on inference chain distillation, comprising the following steps: Step S1: Encode the pre-collected and stored four diagnostic modal data into four modal projection vectors respectively. Then, perform a weighted summation of the four modal projection vectors according to the modal weight vector to obtain the shared diagnostic feature vector. Step S2: Input the shared dialectical feature vector into the teacher model and the student model respectively. The teacher model and the student model include multiple inference stages. The decoder of each inference stage generates the hidden state of the current stage based on the hidden state of the previous stage and the shared dialectical feature vector. The decoder of each inference stage uses the hidden state of the current stage as the query and the four modal projection vectors as the key to perform cross attention, and obtains the cross attention weight of the current stage on the four modal projection vectors. Based on the hidden state of the current stage and the cross attention weight, calculate the transition probability distribution of the current stage on the three actions of forward, backward and termination. Step S3: When the probability of terminating the action is the highest, stop the reasoning and output the structured dialectical reference information of the completed reasoning stage; when the probability of backtracking is greater than the probability of advancing and the number of backtrackings has not reached the upper limit, update the modal weight vector according to the cross-attention weight of this stage, recalculate the shared dialectical feature vector and re-execute the reasoning from the previous stage; otherwise, move to the next stage. Step S4: Align the transition probability distributions of the teacher model and the student model, complete the inference chain distillation, and output structured dialectical reference information from the student model.

[0007] Preferably, the step S1 of encoding the four diagnostic modal data into modal projection vectors includes the following steps: the four diagnostic modal data includes visual diagnosis image data, auscultation and olfaction voice data, inquiry and consultation text data, and palpation pulse data; the encoding is as follows: the modal feature vectors of the visual diagnosis image data, auscultation and olfaction voice data, inquiry and consultation text data, and palpation pulse data are extracted by a modal encoder, and each modal feature vector is mapped to a shared diagnostic feature space and aligned across modal semantics to obtain four modal projection vectors; each component of the modal weight vector is non-negative and sums to 1, the initial value of the modal weight vector at the beginning of an inference is equal weight for each modality, and when there is missing modal data in the four diagnostic modal data, the component corresponding to the missing modal data is set to zero and the remaining components are renormalized.

[0008] Preferably, the transition probability distribution in step S2 is output by a gating function. The gating function takes the hidden state of the current stage and the attention accumulation vector of the current stage as input, and outputs gating weights. The first gating weight of the attention accumulation vector is... m Each component represents the attention head of the decoder in this stage, and each decoder layer's connection to the first... mThe cumulative attention is obtained by summing the cross-attention weights of the modal projection vectors. The gating weights are normalized to obtain the transition probability distribution of the three actions of forward, backward, and termination in this stage. The multiple reasoning stages are sequentially connected: the four diagnostic methods analysis stage, the pathogenesis identification stage, the syndrome differentiation stage, the treatment method establishment stage, and the prescription organization stage. The four diagnostic methods analysis stage uses the shared syndrome differentiation feature vector as input to extract tongue color features, pulse features, cold and heat features, deficiency and excess features, and exterior and interior features to obtain a set of syndrome differentiation elements. The pathogenesis identification stage calculates the etiology, location, nature, and progression of the disease based on the set of syndrome differentiation elements and outputs the pathogenesis probability score. The process involves several stages: The syndrome type determination stage, based on the hidden state from the pathogenesis identification stage, calculates the probability distribution of each syndrome type in a predefined syndrome type set using a softmax classifier, and selects the syndrome type with the highest probability as the syndrome type determination result; the treatment method establishment stage generates a treatment method strategy vector based on the syndrome type determination result and the pathogenesis probability distribution vector; the prescription organization stage retrieves candidate prescription information from the prescription database based on the treatment method strategy vector, adjusts the candidate prescription information according to individual difference parameters, and outputs prescription reference information including the proportions of medicinal ingredients and dosages, which serves as candidate information awaiting review by clinicians.

[0009] Preferably, the gating function in step S2 also takes the normalized entropy of the output distribution of this stage as input; the structured dialectical reference information includes prompt information for modal data to be supplemented, the modal data to be supplemented is the modal data corresponding to the modality with the lowest cumulative attention in this stage, and the cumulative attention is the amount obtained by summing the cross-attention weights of each attention head of the decoder and each decoder layer on the corresponding modal projection vector in this stage; after the probability of the backtracking action in the first inference stage among the multiple inference stages is set to zero, the probabilities of the remaining actions are renormalized.

[0010] Preferably, updating the modality weight vector according to the cross-attention weights in step S3 includes the following steps: adjusting the attention heads and decoder layers of the decoder in this stage to the first... m The cross-attention weights of the modal projection vectors are summed to obtain the first... m The cumulative attention value of each modality is normalized by softmax with a temperature coefficient, and then used as the updated modality weight vector. The temperature coefficient is 0.2 to 1.0, and the upper limit is 1 to 3 times. When the increment of the maximum probability of the stage conclusion output after re-execution of inference in this stage is less than the confidence gain threshold, the process moves to the next stage. The stage conclusion probability distribution is the normalized distribution of the decoder output in this stage, and the confidence gain threshold is 0.01 to 0.05.

[0011] Preferably, the modal encoder includes: a visual Transformer encoder for extracting the modal feature vectors of the visual image data; a speech Transformer encoder for extracting the modal feature vectors of the auscultation speech data; a text Transformer encoder for extracting the modal feature vectors of the inquiry text data; and a one-dimensional residual convolutional network encoder for extracting the modal feature vectors of the palpation pulse data. The vector output by the one-dimensional residual convolutional network encoder after multiple layers of one-dimensional convolution and residual connections is concatenated with the heart rate variability features calculated from the palpation pulse data to obtain the modal feature vectors of the palpation pulse data. The cross-modal semantic alignment adopts a contrastive learning loss function, using different modal data of the same patient to form positive sample pairs and modal data of different patients to form negative samples, so that the modal projection vectors of the same patient are mutually aggregated in the shared diagnostic feature space.

[0012] Preferably, the total loss of the inference chain distillation in step S4 is a weighted sum of the state distillation loss, the transition distillation loss, and the decision distillation loss according to the weight coefficients. The state distillation loss is as follows: the hidden states of the teacher model at each inference stage are projected to the same dimension space as the hidden states of the student model through a linear projection layer. The projected hidden state set of the teacher model and the hidden state set of the student model within the same training batch are used as two sample sets. The maximum mean difference between the two sample sets at each inference stage is measured by a Gaussian kernel and summed over each inference stage. The transfer distillation loss is the sum of the KL divergence between the transfer probability distributions of the teacher model and the student model at each inference stage at each inference stage; The decision distillation loss is the sum of the cross-entropy loss calculated by scaling the logarithmic probabilities of the proof type output by the teacher model and the student model using the distillation temperature, and the mean squared error loss that constrains the alignment of the gating weights used by the teacher model and the student model to generate the transition probability distribution at each inference stage.

[0013] Preferably, the inference chain distillation is performed sequentially through four training phases according to a progressive training strategy: The first training phase distills the reasoning chain knowledge from the first reasoning phase to the second reasoning phase; The second training phase adds distillation for the third inference phase on top of freezing the parameters trained in the first training phase. The third training phase adds distillation for the fourth inference phase based on freezing the parameters trained in the first and second training phases; The fourth training phase adds distillation for the fifth inference phase based on freezing the trained parameters from the first, second, and third training phases. The total loss for each training phase is a weighted sum of the state distillation loss, the transition distillation loss, and the decision distillation loss corresponding to the activated inference phase.

[0014] Preferably, the structured dialectical reference information in step S4 includes: the evidence type determination result and the confidence probability distribution corresponding to the evidence type determination result; the transition probability distribution of each reasoning stage; and the actual transition path of the reasoning chain, wherein the actual transition path includes the sequence number of the reasoning stage in which the backtracking action occurs, the value of the modal weight vector before and after the backtracking action, and the confidence probability distribution before and after the backtracking action.

[0015] This invention also provides a multimodal data processing system for TCM syndrome differentiation based on inference chain distillation, the system comprising: Multimodal data acquisition module: used to acquire pre-collected and stored four diagnostic modal data, including visual diagnosis image data, auscultation and olfaction voice data, inquiry text data, and palpation pulse data; Modality encoding and alignment module: used to encode the four diagnostic modal data into modality projection vectors, and to obtain a shared diagnostic feature vector by weighting and summing the modality projection vectors according to the modality weight vector; The chain-based reasoning module includes a teacher model and a student model, each with multiple reasoning stages. The decoder at each stage generates the hidden state of the current stage from the hidden state of the previous stage and the shared dialectical feature vector. Based on the hidden state of the current stage and the cross-attention weights of the decoder's modal projection vector, a gating function is used to calculate the transition probability distribution for forward, backward, and termination actions in the current stage. When the probability of the termination action is maximum, the chain-based reasoning module stops reasoning and outputs structured dialectical reference information for the completed reasoning stage and a prompt for supplementary modal data. When the probability of the backward action is greater than the probability of the forward action and the number of backward actions has not reached the upper limit, the chain-based reasoning module sends the cross-attention weights of the current stage to the modal encoding and alignment module. The modal encoding and alignment module updates the modal weight vector and recalculates the shared dialectical feature vector. The chain-based reasoning module then re-executes reasoning from the previous stage using the recalculated shared dialectical feature vector; otherwise, it moves to the next stage. Inference chain distillation module: used to align the transition probability distributions of the teacher model and the student model with KL divergence to complete inference chain distillation, and the student model outputs structured dialectical reference information.

[0016] The advantages of this invention include at least the following: 1. The TCM diagnosis process is modeled as a multi-stage chain reasoning model. Each stage generates a transition probability distribution on the three actions of advancing, retreating, and terminating through a gating function, so that the diagnosis process is structured and interpretable. The model not only outputs structured diagnosis reference information, but also fully displays the reasoning path, the stage where retreat occurs, and the changes in modal weights and confidence before and after retreat. 2. By using a modal encoder and a linear projection function, four heterogeneous modal data are mapped to a shared diagnostic feature space. A contrastive learning loss function is used to achieve cross-modal semantic alignment, so that features pointing to the same syndrome in different modalities are automatically clustered in the shared space, thus solving the semantic gap problem of the four diagnostic data. 3. Misleading evidence of low quality or weakly related modalities is reduced in weight after being detected by the downstream stage. The recalculated shared dialectical feature vector changes the results of dialectical element extraction in the upstream stage, so that misleading evidence is no longer continuously amplified along the chain. 4. The reasoning chain has a transition probability distribution consisting of three actions: forward, backward, and termination at each stage. The path decision of the teacher model in the dialectical process becomes a measurable and alignable distribution, which can be transferred to the student model as a transfer distillation loss through KL divergence. The student model not only learns the teacher's final conclusion and intermediate representation, but also learns the judgment of the teacher that should be reanalyzed under what evidence conditions, so that the lightweight model can maintain the robustness of the dialectical path under the condition of low-quality data collection at the grassroots level. Attached Figure Description

[0017] Figure 1 This is a schematic diagram of the method flow according to an embodiment of the present invention; Figure 2 This is a schematic diagram of the stage transition decision and rollback process according to an embodiment of the present invention; Figure 3 This is a heatmap showing the similarity of hidden states and the consistency of stage transition probabilities between the teacher model and the student model in an embodiment of the present invention. The left figure shows the similarity of the reasoning states of the teacher model and the student model at each reasoning stage, and the right figure shows the consistency of the stage transition probability distribution between the teacher model and the student model at each reasoning stage. Figure 4 This is a schematic diagram of the system structure according to an embodiment of the present invention. Detailed Implementation

[0018] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the protection scope of the present invention.

[0019] like Figure 1 As shown, this embodiment of the invention provides a method for processing multimodal data of TCM syndrome differentiation based on inference chain distillation, including the following steps: Step S1: Encode the pre-collected and stored four diagnostic modal data into four modal projection vectors respectively. Then, perform a weighted summation of the four modal projection vectors according to the modal weight vector to obtain the shared diagnostic feature vector.

[0020] This step extracts feature vectors from four different modalities of clinical data and maps them to a unified semantic space. In this embodiment of the invention, the four diagnostic modalities include visual examination image data, auscultation and olfaction voice data, inquiry text data, and palpation pulse data, each of which is used for feature extraction through its corresponding modality encoder.

[0021] Tongue images were collected during visual diagnosis. The raw data consisted of a three-dimensional tensor containing the image's pixel height, width, and RGB color channels. In the preprocessing stage, the image was first geometrically aligned and cropped based on key points of the mouth contour. Then, color thresholding and morphological operations were used to extract the region of interest (ROI) on the tongue surface, eliminating interference from the lips and teeth. The preprocessed tongue image was input into a pre-trained visual Transformer encoder. This encoder divided the image into fixed-size image blocks. Each image block was linearly projected to obtain a vector representation, which was then superimposed with positional encoding and input into a multi-layer Transformer encoder. The final output vector corresponding to the classification label position is the global feature representation of the tongue image. This feature vector fully encodes the visual features of tongue color, shape, and coating. Tongue color includes red, light red, bluish-purple, and pale white, reflecting the state of Qi and blood and the nature of the disease (hot or cold). Tongue shape includes large, thin, cracked, and teeth-marked, reflecting the functional state of the internal organs. Tongue coating includes white, yellow, grayish-black, greasy, and dry, reflecting the nature and depth of the pathogenic factors. In this embodiment, the visual Transformer encoder divides the image into 16x16 pixel image blocks, and outputs a 768-dimensional modal feature vector after passing through 12 layers of Transformer encoders.

[0022] Auscultation and olfaction are used to collect the patient's speech audio signals. In the preprocessing stage, the audio is digitized at a sampling rate of 16kHz, and an 80-dimensional Mel-ray spectrogram is calculated after removing silence segments using a speech activity detection algorithm. The preprocessed spectrum is input into a pre-trained speech Transformer encoder; in this embodiment, the Whisper model is used. This model employs an encoder-decoder structure and outputs a 768-dimensional modal feature vector. In the semantics of auscultation and olfaction in Traditional Chinese Medicine, hoarseness may indicate lung and kidney yin deficiency, a weak voice may indicate qi deficiency, and wheezing and shortness of breath may indicate lung qi reversal.

[0023] The process involves collecting textual descriptions from patients during a consultation. The preprocessing stage first standardizes the original text using Traditional Chinese Medicine (TCM) terminology, mapping non-standard expressions to the standard TCM terminology system; for example, the non-standard expression "stomach ache" is mapped to the standard term "abdominal pain." Then, a byte-pair encoding word segmenter is used to divide the text into sub-word units, and classification tags are appended to the beginning of the sequence. The preprocessed sequence is then input into a pre-trained text Transformer encoder; in this embodiment, the RoBERTa model is used. The 768-dimensional vector corresponding to the classification tag position represents the global semantic representation of the consultation text.

[0024] Pulse signals were collected from patients via palpation using an electronic pulse oximeter at a sampling frequency of 1 kHz or higher. In the preprocessing stage, median filtering was used to remove impulse noise, followed by wavelet transform denoising. In this embodiment, a db4 wavelet basis was used for 4-layer decomposition to remove high-frequency background noise. Subsequently, the temporal and frequency domain features of heart rate variability were calculated, comprising 6 dimensions in this embodiment: mean RR interval, SDNN, RMSSD, pNN50, low-frequency power (LF), and the low-frequency / high-frequency power ratio (LF / HF). The preprocessed signal was input into a one-dimensional residual convolutional network encoder, which contained multiple layers of one-dimensional convolutions and residual connections, with a network depth of 18 to 50 layers. The final global average pooling layer aggregated the temporal features into a 512-dimensional vector. This 512-dimensional vector was concatenated with the aforementioned 6-dimensional heart rate variability features to obtain a 518-dimensional palpation modal feature vector. Traditional Chinese medicine pulse diagnosis is divided into 28 basic pulse types, including floating pulse, deep pulse, slow pulse, rapid pulse, slippery pulse, hesitant pulse, weak pulse, and strong pulse, each corresponding to a different pathological state.

[0025] The feature vectors of the four modalities come from different feature spaces and cannot be directly compared or fused. However, at the semantic level of Traditional Chinese Medicine (TCM) diagnosis, data from different modalities may point to the same syndrome. For example, a red tongue in tongue appearance, a high-pitched and rapid voice, a bitter taste in the mouth and dry throat in text description, and a rapid pulse in pulse diagnosis all point to a heat syndrome in TCM semantics. This embodiment of the invention uses a modality-specific linear projection function to map the feature vectors of each modality to a unified shared diagnostic feature space. Taking the observation feature as an example, the projection process involves left-multiplying the modality feature vector by a projection weight matrix and adding a bias vector to obtain the observation modality projection vector. The other three modalities are mapped similarly using independent projection parameters. In this embodiment, the shared diagnostic feature space has a dimension of 256. The projection weight matrices for the observation, auscultation, and inquiry modalities are all 256 x 768, while the projection weight matrix for the palpation modality is 256 x 518. The projected vectors of the four modalities all belong to the same dimensional space, within which distance measurements can be performed.

[0026] To ensure semantic alignment of the projected vectors, this embodiment of the invention employs the InfoNCE contrastive learning loss function for cross-modal semantic alignment. Data from different modalities of the same patient constitutes positive sample pairs, while data from different patients constitutes negative samples. The cosine similarity of the positive sample pair in the shared dialectical feature space is scaled using a temperature coefficient, and the resulting exponent is divided by the sum of the corresponding exponent terms of all samples in that modality within the same batch. The negative logarithm is then taken to obtain the InfoNCE loss value for that positive sample pair. In this embodiment, the temperature coefficient is set to 0.07, and the contrastive learning batch size is set to 64. This contrastive learning batch size is the batch size for the cross-modal semantic alignment pre-training stage, distinct from the distillation training batch size for the inference chain distillation stage described later. During training, positive sample pairs are randomly sampled from six combinations: observation and auscultation, observation and inquiry, observation and palpation, auscultation and inquiry, auscultation and palpation, and inquiry and palpation. The InfoNCE loss is calculated for each combination and then averaged. After training convergence, the four modal projection vectors of the same patient are automatically aggregated in the shared space, achieving cross-modal semantic alignment of the four diagnostic data.

[0027] The formula for calculating the InfoNCE loss mentioned above is shown below: ; in, For cross-modal alignment loss; This is a 256-dimensional latent vector of a certain mode after projection; The 256-dimensional latent vector of another modality of the same patient after projection, and Forming positive sample pairs; For the first batch The latent vector of the modality corresponding to each sample is projected, and the summation term iterates through the positive samples. Compared with all negative samples, the patients in question are... Different samples are negative samples; cos(·,·) is the cosine similarity; For comparison, the temperature coefficient is set to 0.07; N To compare the learning batch size, a value of 64 is used.

[0028] After alignment, this embodiment of the invention uses the modal weight vector to weight and sum the four modal projection vectors to obtain the shared dialectical feature vector. The shared dialectical feature vector is equal to the sum of the following four terms: the visual inspection weight multiplied by the visual inspection projection vector, the auscultation and olfaction weight multiplied by the auscultation and olfaction projection vector, the inquiry weight multiplied by the inquiry and olfaction projection vector, and the palpation weight multiplied by the palpation projection vector. Each component of the modal weight vector is non-negative and sums to 1. At the start of an inference, its initial value is equal for each modality, i.e., all four components are 0.25. When data for one or more modalities is missing, the weights of the missing modalities are reset to zero, and the remaining components are renormalized. For example, when palpation pulse data is missing, the initial weights of the visual inspection, auscultation and olfaction, and inquiry modalities are all one-third. It should be noted that the modal weight vector is not constant during an inference process. When the gating function of a certain inference stage determines that a backtracking action should be performed, the cross-attention weights of the decoder for each modal projection vector in that stage will be used to redetermine the value of the modal weight vector and recalculate the shared dialectical feature vector accordingly.

[0029] Step S2: Input the shared dialectical feature vector into the teacher model and the student model respectively. Both the teacher model and the student model include multiple inference stages. The decoder of each inference stage generates the hidden state of the current stage based on the hidden state of the previous stage and the shared dialectical feature vector. The decoder of each inference stage uses the hidden state of the current stage as the query and the four modal projection vectors as the key values ​​to perform cross attention, and obtains the cross attention weights of the four modal projection vectors of the current stage. Based on the hidden state of the current stage and the cross attention weights, calculate the transition probability distribution of the current stage in the three actions of forward, backward and termination. Step S3: When the probability of terminating the action is the highest, stop the reasoning and output the structured dialectical reference information of the completed reasoning stage; when the probability of backtracking is greater than the probability of advancing and the number of backtrackings has not reached the upper limit, update the modal weight vector according to the cross-attention weight of this stage, recalculate the shared dialectical feature vector and re-execute the reasoning from the previous stage; otherwise, move to the next stage.

[0030] This invention models the TCM diagnostic reasoning process as a five-stage chain-like reasoning process, with each stage corresponding to a key step in TCM diagnostic reasoning. The formal expression of the reasoning chain is as follows: i The hidden state of the first stage is equal to the hidden state of the second stage. i The Transformer decoder in this stage operates on the hidden state and the result of the shared dialectical feature vector from the previous stage, where... i Choose from 1 to 5, and take the shared dialectical feature vector for the hidden state in stage 0.

[0031] The formal expression of the above reasoning chain is shown in the following equation: ; ; in For the first i The hidden states of each stage, i Take 1 to 5; This is the hidden state from the previous stage. Take the shared dialectical feature vector; A Transformer decoder shared across all inference stages; For the first i The stage embedding vectors are used to distinguish different inference stages. The shared decoder uses cross-attention to combine the hidden state of the previous stage with the shared dialectical feature vector. Integration and integration of stages The hidden state of the production cost stage; For the first Modal projection vectors, Take 1 to 4; This indicates the hidden state at this stage. For querying, the cross-attention function with four modal projection vectors as keys outputs... For this stage, the first Cross-attention weights of each modal projection vector; To share the dialectical feature vector, it is equal to the weighted sum of the four modal projection vectors according to the modal weight vector.

[0032] Hidden state generation and modal cross-attention are two sequentially executed steps: the key value of the former is the shared dialectical feature vector. The latter's key value is four modal projection vectors. to The two have different key-value sources.

[0033] The first stage is the four diagnostic methods analysis stage, which uses the shared diagnostic feature vector as the initial input to extract the feature dimensions directly related to syndrome determination, including tongue color features, pulse features, and cold / heat, deficiency / excess, and exterior / interior features, forming a preliminary set of diagnostic elements.

[0034] The second stage is the pathogenesis identification stage, which deduces the internal pathogenesis of the disease based on the results of the four diagnostic methods, including the four aspects of etiology, location, nature and progression of the disease, and outputs the pathogenesis probability distribution vector.

[0035] The third stage is the syndrome differentiation stage. A softmax classifier is used to calculate the probability distribution of each syndrome in a predefined set of syndromes, and the syndrome with the highest probability is selected as the determination result. Simultaneously, concurrent syndrome relationships are identified. The predefined set of syndromes is constructed based on the national standard for syndrome differentiation in TCM clinical diagnosis and treatment terminology and the disease range of this embodiment. This embodiment includes a total of 12 syndromes.

[0036] The fourth stage is the treatment method establishment stage, where the corresponding treatment method is derived based on the syndrome differentiation results and the pathogenesis probability distribution vector. Specifically, the hidden state from the syndrome differentiation stage is concatenated with the pathogenesis probability distribution vector and input into the decoder of this stage. The decoder outputs a treatment strategy vector, which is 128-dimensional in this embodiment. This vector is then processed by a treatment method classification head to obtain the treatment method probability distribution on a predefined set of treatment method labels. The treatment method label corresponding to the highest probability is selected as the primary treatment method. During training, treatment method labels marked by renowned traditional Chinese medicine practitioners are used as supervision signals, and cross-entropy loss is employed. The above process embodies the core principle of syndrome differentiation and treatment in traditional Chinese medicine: syndrome differentiation determines treatment method.

[0037] The fifth stage is the prescription organization stage. Based on the treatment strategy vector, prescriptions matching the treatment method are retrieved from the prescription database. Adjustments are made to the prescriptions according to the patient's individual differences, outputting prescription reference information including the ingredients and dosage ratios. The prescription database is indexed by treatment method tags. Each record includes the prescription name, constituent ingredients, baseline dosage, efficacy classification, and corresponding treatment method tag. In this embodiment, 152 prescriptions are included, covering 12 categories of treatment method tags: kidney tonification, liver soothing, phlegm resolution, and blood circulation promotion. During retrieval, the prescriptions are sorted by the cosine similarity between the treatment strategy vector and the embedding vector of each prescription's treatment method tag, and the highest similarity is selected as the candidate prescription information. Individual difference parameters include patient age, constitution type, and concurrent symptoms. Adjustments are made according to an ingredient addition / subtraction rule table, which uses a triplet format of concurrent symptoms, ingredient addition / subtraction, and dosage adjustment. In this embodiment, there are a total of 42 addition / subtraction rules. The adjustment of the baseline dose based on body constitution type shall be performed according to the body constitution type dose coefficient, with 1.0 to 1.2 for Qi deficiency and Yang deficiency constitution, and 0.8 to 1.0 for Yin deficiency constitution.

[0038] In this embodiment of the invention, the teacher model shares a six-layer Transformer decoder with a hidden dimension of 1024 and 16 attention heads at each inference stage; the student model shares a two-layer Transformer decoder with a hidden dimension of 256 and 4 attention heads at each inference stage; each inference stage is distinguished by a stage embedding vector. Both the teacher and student models' decoders fuse the hidden state of the previous stage with the shared dialectical feature vector through a cross-attention mechanism.

[0039] The gating function for each stage is a two-layer fully connected network with a hidden layer dimension of 128, an activation function of ReLU, and an output layer dimension of 3. The input to the gating function is a concatenation of the following three parts. The first part is the hidden state of the current stage, which is the average pooling vector of the last hidden layer of the decoder in that stage. The second part is the 4-dimensional attention cumulative vector formed by the cross-attention weights of the decoder's projections onto the four modalities in the current stage. m Each component represents the current hidden state of the query end and the key-value end of each layer and attention head of the decoder at the current stage. mThe attention weights of the modal projection vectors are summed in the layer and head dimensions to obtain the first... m The attention accumulation of each modality, together with the four accumulations, constitutes a 4-dimensional attention accumulation vector. This 4-dimensional vector represents which modal evidences were actually relied upon and to what extent during the current reasoning stage when reaching its conclusion. The third part is the normalized entropy of the current stage's output distribution. For the pathogenesis identification stage, it's the entropy of the pathogenesis probability distribution vector; for the syndrome determination stage, it's the entropy of the syndrome probability distribution; and for other stages, it's the entropy of the output vector after softmax. The entropy value is divided by the logarithm of the support set size of the distribution to normalize it to the 0-1 range. The physical meaning of the normalized entropy is the degree of hesitation in the current stage's conclusion. The closer the entropy value is to 1, the more difficult it is to choose among multiple candidate conclusions, and the more likely the evidence relied upon in this stage is contradictory or insufficient.

[0040] In another implementation, the input to the gating function is only the concatenation of the average pooling vector of the hidden state at the current stage and the 4-dimensional attention accumulation vector. In this case, the gating function judges the transition action based on the distribution of modal evidence actually relied upon at the current stage, and the output layer dimension is still 3. Compared with the implementation that simultaneously inputs normalized entropy, this implementation has a smaller number of gating function parameters and is suitable for deployment scenarios with stricter requirements for inference latency.

[0041] The gating function outputs 3D gating weights, which, after softmax normalization, yield the transition probability distribution for the three transition actions—forward, backward, and termination—in the current stage. For the four diagnostic stages, since there is no previous stage, the probability of the backward action is set to zero, and the other two components are renormalized.

[0042] When the probability of moving forward is at its maximum, the inference chain moves from the current stage to the next stage, and the hidden state of the current stage becomes the input to the decoder of the next stage.

[0043] When the probability of a backtracking action is greater than the probability of a forward action and the number of backtracking attempts has not reached the upper limit, the following process is executed. First, the modal weight vector is updated: the cross-attention weights of the current stage decoder for each modality projection vector are summed at each attention head and each decoder layer to obtain the cumulative attention for each modality and form a 4-dimensional cumulative attention vector. Then, softmax normalization with a temperature coefficient is applied to each component of this vector. The result is the updated modal weight vector, i.e., the updated modal weight vector. m The weight of the mode is equal to the _th_ modality mThe cumulative attention value of each modality is divided by the temperature coefficient, and then the exponent is taken. This exponent is then divided by the sum of the exponents of the four modalities. The temperature coefficient ranges from 0.2 to 1.0, and is 0.5 in this embodiment. The smaller the temperature coefficient, the sharper the weight distribution, and the lower the weight of the modality ignored in the current stage is. The larger the temperature coefficient, the smoother the weight distribution, and the smaller the deviation from the original equal weight distribution. If the data for a certain modality is missing, the cumulative attention value corresponding to that modality is set to negative infinity when calculating the softmax, which is equivalent to setting its exponent to zero, so that the weight of that modality is zero before normalizing the other modalities. Then, the shared dialectical feature vector is recalculated, and the four modal projection vectors are re-weighted and summed using the updated modal weight vector. Finally, inference is re-executed from the previous stage of the current stage, that is, the recalculated shared dialectical feature vector is used as the decoder input to regenerate the hidden state of the previous stage, and then the hidden state and transition probability distribution of the current stage are regenerated, and the backtracking count is incremented by 1.

[0044] The formula for calculating the modal weight update is shown below: ; in, The updated modal weight vector is the first... The component, i.e., the updated component. The weights of each modality; For the first Cumulative attention across modalities; The temperature coefficient of the softmax function; The modal number is 1 to 4.

[0045] When the probability of terminating the action is at its highest, the reasoning stops, the structured dialectical reference information of the completed stage is output, and a prompt message for supplementary modality data is output. The modality to be supplemented, as indicated in the prompt message, is the modality with the lowest cumulative attention value in the current stage's 4-dimensional attention cumulative vector, meaning that almost no usable evidence can be obtained from this modality in the current stage.

[0046] To prevent the inference chain from repeatedly looping between backtracking and re-execution, this embodiment of the invention sets two termination conditions, one of which forces a forward action. The first termination condition is an upper limit on the number of backtracks. When the total number of backtracks in a single inference reaches this upper limit, a forward action is forced. The upper limit for the number of backtracks ranges from 1 to 3; in this embodiment, it is 2. A value that is too small is insufficient to correct misleading modalities, while a value that is too large causes a linear increase in inference latency. The second termination condition is a confidence gain threshold. When the increment of the maximum probability of the output distribution at the current stage after a backtrack and re-execution is less than the confidence gain threshold, it indicates that the current modality weight update can no longer provide further evidence improvement, and a forward action is forced. The confidence gain threshold ranges from 0.01 to 0.05; in this embodiment, it is 0.02. These two conditions together ensure that the number of decoder forward passes in a single inference has a definite upper bound, thus ensuring bounded inference latency.

[0047] like Figure 2 As shown, the above-described stage transition decision and rollback process constitute a complete decision path. After generating the hidden state at each stage, the ternary transition probability distribution is calculated, and the action branch with the highest probability is executed. The rollback branch returns to the previous stage after updating the modal weights and recalculating the shared dialectical feature vector. The termination branch outputs the results of the completed stage and the modal hints to be supplemented. Two termination conditions force the rollback branch to switch back to the forward branch. This process has full coverage of branch conditions, no dead zones, and uses the forward action as the default path.

[0048] To prevent inference chain breaks or contradictions, this embodiment of the invention also introduces an inference chain consistency verification mechanism. During distillation training, a semantic implication model fine-tuned on a Chinese natural language inference dataset is used to determine the implication relationship between inference states in adjacent stages. If the inference states in adjacent stages do not satisfy the semantic implication relationship, a consistency penalty term is applied, with a weight coefficient of 0.2. The input to the semantic implication model is the textual description corresponding to the inference states of two adjacent stages, and the output is a three-class classification result of implication, neutrality, and contradiction. When the output is contradictory, a consistency penalty term is applied. The textual description of the inference state in each stage is obtained by decoding the autoregressive output sequence of the decoder for that stage through a vocabulary. For stages that do not generate natural language sequences, the highest contributing feature dimensions in the hidden state of that stage are restored to dialectical element terms according to a pre-established feature dimension and dialectical element mapping table, and then filled into the fixed description template of that stage to generate a textual description.

[0049] Step S4: Align the transition probability distributions of the teacher model and the student model, complete the inference chain distillation, and output structured dialectical reference information from the student model.

[0050] The teacher model is a large and complex model, while the student model is a lightweight model. The distillation process employs a weighted summation of three layers of loss functions to ensure that the student model learns not only the final output, but also the complete inference path and path decision.

[0051] The first layer represents the state distillation loss. For the first layer in the inference chain... i In each stage, the average pooling vector of the last hidden layer of the Transformer in that stage is extracted as the teacher's inference state representation, and the average pooling vector of the hidden layer of the student model in the corresponding stage is extracted as the student's inference state representation. Since the teacher model has a hidden dimension of 1024 and the student model has a hidden dimension of 256, a learnable linear projection layer is used to project the teacher model's average pooling vector from 1024 dimensions to 256 dimensions before calculating the state distillation loss. Then, using the projected teacher and student state vector sets within the same training batch as two sample sets, a Gaussian kernel is used to calculate the maximum mean difference between them. The kernel bandwidth is set to 1.0, and the distillation training batch size is set to 16. The state distillation loss is obtained by calculating and summing the results for each of the five inference stages.

[0052] The second layer represents transfer distillation losses. For the first layer in the reasoning chain... i In each stage, the KL divergence between the teacher model's ternary transition probability distribution and the student model's ternary transition probability distribution in that stage is calculated. The sum of the KL divergences across the five stages yields the transfer distillation loss. Since the transition probability distribution is defined in the action space comprised of the three actions of advancing, retreating, and terminating, this distribution carries the teacher model's path decision knowledge regarding the evidence state under which it should continue, reverse and start over, or stop and request supplementary evidence. Therefore, this loss transfers the teacher model's reasoning path decisions, not just intermediate representations or the final conclusion. To enable the student model to reproduce the teacher model's retreat behavior, both the teacher and student models use the same gating function structure and the same upper limit on the number of retreats.

[0053] The third layer is the decision distillation loss. The cross-entropy loss is calculated after scaling the log probabilities of the certificate types output by the teacher model and the student model by the distillation temperature. The certificate type probability distribution of the teacher model after temperature transformation is used as the target distribution. The mean square error loss is combined with the alignment of the gating weight vectors of the teacher model and the student model at each stage. The weight coefficient of the gating weight alignment term is 0.5.

[0054] The total distillation loss is the weighted sum of the losses from the three layers mentioned above. The weight coefficient for the state distillation loss is 0.4, the weight coefficient for the transition distillation loss is 0.3, and the weight coefficient for the decision distillation loss is 0.3. When the inference chain consistency check mechanism is enabled, the total distillation loss is the weighted sum of the losses from the three layers above, plus the product of the consistency penalty term and its weight coefficient of 0.2. The consistency penalty term is non-zero only when the semantic implication model output is contradictory, so it does not participate in gradient backpropagation in most training steps.

[0055] This invention employs a progressive training strategy, decomposing the learning process of a complex reasoning chain into four training stages, progressing from simple to complex. Since each reasoning stage shares the same set of decoder parameters, the freezing and activation of parameters in each training stage are stage-specific: the shared decoder parameters are fixed after training in the first training stage, and subsequent training stages involve distillation by introducing stage embeddings and corresponding gating / classification heads for the newly added reasoning stage. The first training stage distills only the reasoning chain knowledge from the four diagnostic methods analysis stage to the pathogenesis identification stage. The second training stage adds distillation for the syndrome differentiation stage based on the frozen parameters trained in the first training stage. The third training stage adds distillation for the treatment method establishment stage based on the frozen parameters trained in the aforementioned training stages. The fourth training stage adds distillation for the prescription organization stage based on the frozen parameters trained in the aforementioned training stages, completing the five-stage distillation of the entire chain. The total loss for each training stage is the weighted sum of the state distillation loss, transition distillation loss, and decision distillation loss corresponding to the activated reasoning stage; the loss terms for the inactive stages are not included in the calculation. Simultaneously, a distillation temperature decay strategy is employed, with the distillation temperature gradually decreasing from 2.0 to 1.0, and the learning rate decreasing to one-fifth of the previous training stage for each subsequent training phase.

[0056] Example 1 This embodiment uses the clinical experience of renowned traditional Chinese medicine (TCM) doctors in treating ovulatory infertility as its subject, demonstrating the entire reasoning process under the conventional circumstance of consistent and sufficient evidence from the four diagnostic methods. First, 860 complete medical records of patients with ovulatory infertility treated by renowned TCM doctors were collected. Each record included images of tongue appearance from observation, audio recordings of auscultation and olfaction, textual records of questioning, and pulse signals from palpation. All clinical medical record data used in this embodiment had obtained informed consent from patients and passed ethical review, and underwent de-identification processing before use.

[0057] The data annotation adopts a human-computer collaborative annotation method. First, the large language model automatically generates preliminary inference chain annotations based on the five-stage inference chain annotation framework. Then, three attending physicians from the team of renowned traditional Chinese medicine doctors review and correct the annotation results of each inference stage, and finally construct a dataset containing 860 complete dialectical inference chains.

[0058] The teacher model employs a five-stage cascaded Transformer decoder autoregressive sequence generation architecture. The five inference stages share the same six-layer Transformer decoder, distinguished by stage embedding vectors. It has a hidden dimension of 1024 and 16 attention heads. The teacher model is trained using a dataset of 860 inference chains, employing the AdamW optimizer with a learning rate of 2 x 10^-4, a distillation training batch size of 16, and a training duration of 50 epochs. To enhance the model's robustness to modality loss, a modality dropout data augmentation strategy is used during training, randomly setting the projection vector of one modality to zero with a probability of 0.15 for each training sample. The student model uses the same five-stage cascaded architecture as the teacher model, but the decoder in each stage is reduced to two Transformer layers, with a hidden dimension of 256 and 4 attention heads. The weight coefficients for distillation training are 0.4, 0.3, and 0.3, respectively, while the weight coefficient for the consistency check penalty term is 0.2. In the progressive distillation training, the first training phase consists of 20 epochs, the second training phase consists of 15 epochs, and the third and fourth training phases each consist of 10 epochs.

[0059] With the above configuration, the total number of parameters in the five-stage chained inference decoder, gating function, and modal projection layer of the teacher model is approximately 120M. The pre-trained encoders for each modality are not included in the statistics, and the five inference stages share the same set of decoder parameters. The single inference latency is approximately 150ms, and the GPU memory usage is approximately 2048MB. After inference chain distillation, the total number of parameters in the student model is compressed to approximately 7.5M, the single inference latency is reduced to approximately 30ms, and the GPU memory usage is only approximately 128MB. The parameter compression ratio reaches 93.8%, the inference latency is reduced by approximately 80%, and the GPU memory usage is reduced by approximately 94%, meeting the real-time deployment needs of primary healthcare institutions on conventional computing devices.

[0060] The deployed student model reasoned step-by-step based on the four diagnostic methods (inspection, auscultation and olfaction) of a 35-year-old female infertility patient. The modal weight vector was initialized with an equal weight of 0.25. The outputs of each stage are as follows: In the four diagnostic methods analysis stage, the model extracted features of a pale red tongue and thin white coating from observation, and a wiry and thready pulse from palpation, suggesting signs of kidney deficiency and liver stagnation. The confidence level for this stage was 0.88, and the cumulative attention vectors were: observation 0.28, auscultation 0.14, inquiry 0.30, and palpation 0.28. The normalized entropy was 0.31, and the transition probability distribution was: forward 0.94, termination 0.06. The forward action was executed. In the pathogenesis identification stage, the dominant pathogenesis was identified as kidney deficiency with a probability weight of 0.48, and the associated pathogenesis was identified as liver stagnation with a probability weight of 0.32, phlegm-dampness 0.12, and blood stasis 0.08. The normalized entropy was 0.85, and the transition probability distribution was: forward 0.89, backward 0.07, and termination 0.04. The forward action was executed. In the syndrome differentiation stage, the syndrome was determined to be kidney deficiency with liver stagnation. The confidence level at this stage was 0.82, the normalized entropy was 0.35, and the transition probability distribution was 0.93 (advance), 0.05 (regression), and 0.02 (termination). The "advance" action was executed. In the treatment principle establishment stage, the treatment principle was determined to be tonifying the kidney, soothing the liver, regulating menstruation, and promoting fertility. The "advance" action was executed. In the prescription organization stage, the prescription included Angelica sinensis 15g, Paeonia lactiflora 12g, Bupleurum chinense 10g, Rehmannia glutinosa 15g, Cuscuta chinensis 12g, Dipsacus asper 10g, and Cyperus rotundus 10g, as candidate information for clinicians' reference.

[0061] In this embodiment, the quality of the four diagnostic methods data is good, the confidence probability distribution is higher than 0.8, the gating functions of all five stages output the highest probability of advancing, the inference chain advances to the end without backtracking, and the modal weight vectors always maintain equal weight. This embodiment illustrates that under normal circumstances where the evidence from the four diagnostic methods is consistent and sufficient, the backtracking mechanism does not introduce any additional inference delay, and the inference path is completely consistent with the scheme without the backtracking mechanism.

[0062] Example 2 The model, dataset, and training hyperparameters in this embodiment are exactly the same as in Embodiment 1. The difference lies in the fact that the tongue images from the four diagnostic methods of the patients were collected under warm-toned lighting in a primary healthcare facility, resulting in a significant reddish temperature shift. This caused the pale red tongue to be encoded as a visual feature close to a red tongue. The data quality of the other three modalities was normal. The patient was a 41-year-old female. The medical history recorded symptoms of cold limbs, lower back and knee weakness, and delayed menstruation. The pulse upon palpation was deep and thready.

[0063] The modal weight vector is initialized with an equal weight of 0.25. During the four diagnostic methods analysis stage, since the visual diagnosis features are biased towards a red tongue, this stage simultaneously extracts tongue color features that indicate heat and symptoms and pulse features that indicate cold, forming a set of contradictory diagnostic elements. The cumulative attention vector is 0.34 for visual diagnosis, 0.12 for auscultation and olfaction, 0.27 for inquiry, and 0.27 for palpation, with a normalized entropy of 0.58. The transition probability distribution is 0.86 for moving forward and 0.14 for terminating, and the forward action is executed.

[0064] During the pathogenesis identification stage, due to contradictory evidence, the probability distribution of pathogenesis exhibits a clear hesitant characteristic: Kidney Yang Deficiency 0.34, Yin Deficiency with Internal Heat 0.31, Liver Qi Stagnation 0.20, and Phlegm-Dampness 0.15. The probabilities of the two dominant pathogenesis are extremely close. The cumulative attention vector at this stage is 0.14 for observation, 0.10 for auscultation, 0.39 for inquiry, and 0.37 for palpation, with the normalized entropy rising to 0.96. The gating function takes the hidden state at this stage, the cumulative attention vector, and the normalized entropy as input, and outputs a transition probability distribution of 0.21 for moving forward, 0.72 for moving backward, and 0.07 for terminating. The backward action has the highest probability, and the backward action is executed.

[0065] The rollback process is as follows. When updating the modal weight vector, the cumulative attention vectors for the current stage (pathogenesis identification stage) are taken: 0.14 for observation, 0.10 for auscultation, 0.39 for inquiry, and 0.37 for palpation. After dividing by the temperature coefficient of 0.5, softmax normalization is performed, resulting in updated modal weight vectors of approximately 0.19 for observation, 0.18 for auscultation, 0.32 for inquiry, and 0.31 for palpation. It is evident that the pathogenesis identification stage rarely relies on observational evidence when making conclusions; therefore, the weight of the observation modality is significantly reduced, while the weights of inquiry and palpation are increased. Subsequently, the updated modal weight vector is used to re-weight and sum the four modal projection vectors to obtain a new shared diagnostic feature vector, and the reasoning is re-executed from the previous stage (four diagnostic methods analysis stage).

[0066] In the re-execution of the four diagnostic methods analysis stage, the proportion of the visual diagnosis component in the shared diagnostic feature vector decreased from 0.25 to 0.19, and the influence of the reddish tongue color feature on the diagnostic element set was correspondingly weakened. The diagnostic elements extracted in this stage were mainly cold and deficiency signs, and the normalized entropy decreased to 0.39.

[0067] After re-execution, the pathogenesis identification phase's probability distribution became: Kidney Yang Deficiency 0.57, Liver Qi Stagnation 0.18, Yin Deficiency with Internal Heat 0.14, and Phlegm-Dampness 0.11. The dominant pathogenesis was clearly identified, and the normalized entropy decreased to 0.83. The maximum probability in this phase increased from 0.34 to 0.57, an increase of 0.23, which is greater than the confidence gain threshold of 0.02. Furthermore, the number of backtracking steps (1) did not reach the upper limit of 2, therefore, forced forward was not triggered. The transition probability distribution was: Forward 0.91, Backtrack 0.06, Termination 0.03, and the forward action was executed.

[0068] In the syndrome differentiation stage, the syndrome was determined to be kidney yang deficiency with a confidence level of 0.79, and the next action was executed. In the treatment method establishment stage, the treatment method was determined to be warming and tonifying kidney yang, regulating menstruation and promoting fertility. In the prescription organization stage, the candidate prescriptions with the highest matching degree with the warming and tonifying kidney yang treatment method were retrieved from the prescription database, and the dosage coefficient was adjusted from 1.0 to 1.2 according to individual difference parameters, and the rules for adding or subtracting ingredients were applied, and the prescription reference information was output.

[0069] In contrast, if the rollback action is disabled, the pathogenesis identification stage can only proceed under hesitant conditions of Kidney Yang Deficiency (0.34) and Yin Deficiency with Internal Heat (0.31). The syndrome differentiation stage will output the syndrome with low confidence, and the reddish palpable evidence will be transmitted all the way to the treatment method and prescription. If the rollback action is retained but the modal weight vector is not updated, the decoder input will be exactly the same as the previous one when it is re-executed. It will inevitably reproduce the same hidden state, the same pathogenesis probability distribution, and the same transition probability distribution. The rollback will repeat indefinitely until the upper limit of the rollback count is triggered, after which it will be forced to move forward. The result is exactly the same as disabling the rollback action. This embodiment illustrates that the redetering of modal weights and the re-execution of rollback must exist simultaneously. Implementing either step alone will not produce the above-mentioned re-judgment effect.

[0070] Example 3 The model and training hyperparameters in this embodiment are the same as in Embodiment 1. The difference is that the patient only provided visual tongue images and consultation text; the auscultation voice could not be collected due to the noisy consultation environment, and the pulse diagnosis was missing because the primary health center was not equipped with an electronic pulse diagnostic instrument. The patient was a 29-year-old female, and the consultation text only contained two main complaints: irregular menstruation and fatigue, which was relatively low in information content.

[0071] The modal weight vectors were reset to zero for missing modes and then normalized and initialized, i.e., 0.5 for observation, 0 for auscultation and olfaction, 0.5 for inquiry, and 0 for palpation. During the four diagnostic methods analysis phase, only the observation elements of pale tongue and white coating, and the symptom elements of fatigue and menstrual irregularities could be extracted. Pulse evidence was lacking. The cumulative attention vectors were 0.48 for observation, 0.00 for auscultation and olfaction, 0.52 for inquiry, and 0.00 for palpation, with a normalized entropy of 0.71. The transition probability distribution was 0.78 for forward and 0.22 for termination, indicating a forward action was executed.

[0072] During the pathogenesis identification phase, the probability distribution of the four candidate pathogenesis factors is as follows: Qi deficiency 0.29, Kidney deficiency 0.26, Liver stagnation 0.24, and Blood deficiency 0.21. The probabilities of these four factors are close to the average distribution, and the normalized entropy is as high as 0.99. The output transition probability distribution of the gating function is as follows: forward 0.13, backward 0.19, and termination 0.68. The termination action has the highest probability, so the termination action is executed.

[0073] The termination action results in stopping reasoning, outputting structured diagnostic reference information for the completed four diagnostic methods analysis and pathogenesis identification stages, and outputting a prompt message indicating that supplementary modal data is needed. The modal to be supplemented, as indicated in the prompt message, is the modality with the lowest cumulative modal attention, namely auscultation and palpation. The prompt suggests collecting pulse signals and voice recordings before re-inferring the data. The system does not output syndrome determination results, treatment methods, or prescription reference information.

[0074] This embodiment illustrates that when the total amount of evidence from the four diagnostic methods is insufficient to support the diagnosis, regression is ineffective because reallocating the weights of existing modalities when the total amount of evidence is insufficient does not introduce any new evidence. The gating function's choice of termination rather than regression in this evidence state is precisely the technical significance of setting the transition action as a ternary approach (advance, regression, termination) rather than a binary approach. Regression is used to correct misleading evidence in the existing modality, while termination is used to identify insufficient total evidence in the existing modality. The evidence states corresponding to these two actions are different and are jointly determined by the gating function based on the cumulative attention vector and the normalized entropy.

[0075] The method in this invention randomly divides a dataset of 860 inference chains into a training set, a validation set, and a test set in an 8:1:1 ratio, and evaluates the results on the test set. Experimental results show that the student model trained using the inference chain distillation framework achieves an accuracy of 86.3%, and the proportion of inference chain integrity (i.e., all five stages passing the consistency check) exceeds 92%. Compared to the traditional distillation method using only cross-entropy, the accuracy is improved by 8.1%. Compared to the accuracy of 81.2% achieved by distilling all five stages at once, the progressive distillation strategy improves accuracy by 5.1%. Figure 3 As shown, the state similarity between the teacher model and the student model at each reasoning stage is mostly above 0.85, and the consistency of reasoning transition probability is mostly above 0.80, which verifies the effectiveness of knowledge transfer in the reasoning chain.

[0076] like Figure 4 As shown in the figure, this embodiment of the invention also provides a multimodal data processing system for TCM syndrome differentiation based on inference chain distillation, including a processor and a memory, wherein the memory stores a computer program executable by the processor. The system includes a multimodal data acquisition module, a modality encoding and alignment module, a chain inference module, and an inference chain distillation module.

[0077] The multimodal data acquisition module includes a tongue image acquisition device, a voice acquisition device, a consultation text input interface, and an electronic pulse detector, which are used to acquire visual image data, auscultation voice data, consultation text data, and palpation pulse data.

[0078] The modality encoding and alignment module extracts modal feature vectors for each modality using a modality encoder. These feature vectors are then mapped to a shared dialectical feature space using a modality-specific projection function. Cross-modal semantic alignment is performed using the InfoNCE contrastive learning loss function to obtain four modality projection vectors. Finally, the four modality projection vectors are weighted and summed according to their modality weights to obtain the shared dialectical feature vector. This module also receives cross-attention weights from the chained inference module, sums them across each attention head and decoder layer to obtain the cumulative attention for each modality, and updates the modality weight vectors accordingly and recalculates the shared dialectical feature vector.

[0079] The chained reasoning module includes teacher and student models, each with multiple reasoning stages. The decoder in each stage generates the hidden state of the current stage from the hidden state of the previous stage and the shared dialectical feature vector. It then outputs gating weights through a gating function, which are normalized to obtain the transition probability distribution for the three actions: forward, backward, and termination. When the probability of a backward action is greater than the probability of a forward action and the number of backward actions has not reached the upper limit, this module sends the cross-attention weights of the current stage to the modality encoding and alignment module. The modality encoding and alignment module updates the modality weight vector and recalculates the shared dialectical feature vector. The chained reasoning module then re-executes the reasoning from the previous stage using the recalculated shared dialectical feature vector; otherwise, it moves to the next stage.

[0080] The inference chain distillation module is used to transfer the inference chain knowledge from the chain inference module of the teacher model to the student model. It employs a weighted sum of three layers of losses: state distillation loss, transition distillation loss, and decision distillation loss. The transition distillation loss uses KL divergence to measure the difference in transition probability distributions between the teacher and student models at each stage. A progressive training strategy is used to complete the full chain distillation. Finally, the student model outputs structured dialectical reference information containing syndrome type determination results, confidence distribution, inference path, and prescription reference information.

[0081] The method in this embodiment of the invention is executed by a computer. The processing objects of the method are the pre-collected and stored visual examination image data, auscultation and olfaction voice data, inquiry text data, and palpation pulse data. The intermediate inference states, syndrome probability distributions, and prescription reference information output by the student model at each stage are all structured syndrome differentiation reference information, which are provided for clinicians to review, intervene, and refer to at each stage. The final diagnostic conclusion and medication plan are determined by the clinician.

[0082] The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described; only preferred embodiments of the present invention are illustrated. The descriptions are relatively specific and detailed, but they should not be construed as limiting the scope of the present invention. As long as the combination of these technical features does not contradict each other, it should be considered within the scope of this specification.

[0083] It should be noted that those skilled in the art can make various modifications and improvements without departing from the inventive concept, and these all fall within the scope of protection of this invention. Therefore, the scope of protection of this invention should be determined by the appended claims.

Claims

1. A method for processing multimodal data of TCM syndrome differentiation based on inference chain distillation, characterized in that, Includes the following steps: Step S1: Encode the pre-collected and stored four diagnostic modal data into four modal projection vectors respectively. Then, perform a weighted summation of the four modal projection vectors according to the modal weight vector to obtain the shared diagnostic feature vector. Step S2: Input the shared dialectical feature vector into the teacher model and the student model respectively. The teacher model and the student model include multiple inference stages. The decoder of each inference stage generates the hidden state of the current stage based on the hidden state of the previous stage and the shared dialectical feature vector. The decoder of each inference stage uses the hidden state of the current stage as the query and the four modal projection vectors as the key to perform cross attention, and obtains the cross attention weight of the current stage on the four modal projection vectors. Based on the hidden state of the current stage and the cross attention weight, calculate the transition probability distribution of the current stage on the three actions of forward, backward and termination. Step S3: When the probability of terminating the action is the highest, stop the reasoning and output the structured dialectical reference information of the completed reasoning stage; when the probability of backtracking is greater than the probability of advancing and the number of backtrackings has not reached the upper limit, update the modal weight vector according to the cross-attention weight of this stage, recalculate the shared dialectical feature vector and re-execute the reasoning from the previous stage; otherwise, move to the next stage. Step S4: Align the transition probability distributions of the teacher model and the student model, complete the inference chain distillation, and output structured dialectical reference information from the student model.

2. The method for processing multimodal data of TCM syndrome differentiation based on inference chain distillation according to claim 1, characterized in that: The step S1, which encodes the four diagnostic modal data into modal projection vectors, includes the following steps: the four diagnostic modal data includes visual diagnosis image data, auscultation and olfaction voice data, inquiry and consultation text data, and palpation pulse data; the encoding is as follows: the modal feature vectors of the visual diagnosis image data, auscultation and olfaction voice data, inquiry and consultation text data, and palpation pulse data are extracted by a modal encoder, and each modal feature vector is mapped to a shared diagnostic feature space and aligned across modal semantics to obtain four modal projection vectors; each component of the modal weight vector is non-negative and sums to 1, and the initial value of the modal weight vector at the beginning of an inference is equal weight for each modality. When there is missing modal data in the four diagnostic modal data, the component corresponding to the missing modal data is set to zero and the remaining components are renormalized.

3. The method for processing multimodal data of TCM syndrome differentiation based on inference chain distillation according to claim 1, characterized in that: The transition probability distribution mentioned in step S2 is output by a gating function. This gating function takes the hidden state of the current stage and the attention accumulation vector of the current stage as input, and outputs gating weights. The first gating weight of the attention accumulation vector is... m Each component represents the attention head of the decoder in this stage, and each decoder layer's connection to the first... m The cumulative attention amount is obtained by summing the cross-attention weights of the modal projection vectors. The gating weights are normalized to obtain the transition probability distribution of the three actions of forward, backward, and termination in this stage. The multiple reasoning stages are sequentially connected: the four diagnostic analysis stage, the pathogenesis identification stage, the syndrome type determination stage, the treatment method establishment stage, and the prescription organization stage. The four diagnostic analysis stage takes the shared syndrome differentiation feature vector as input and extracts tongue color features, pulse features, cold and heat features, deficiency and excess features, and exterior and interior features to obtain a set of syndrome differentiation elements. The pathogenesis identification stage calculates the etiology, location, nature, and progression of the disease based on the set of syndrome differentiation elements and outputs the pathogenesis probability distribution vector. The syndrome type determination stage calculates the probability distribution of each syndrome type in the predefined syndrome type set based on the hidden state of the pathogenesis identification stage using a softmax classifier and takes the syndrome type with the highest probability as the syndrome type determination result. The treatment method establishment stage generates a treatment strategy vector based on the syndrome type determination result and the pathogenesis probability distribution vector; the prescription organization stage retrieves candidate prescription information from the prescription database based on the treatment strategy vector, and adjusts the candidate prescription information by adding or subtracting according to individual difference parameters, and outputs prescription reference information including the ratio of medicinal ingredients and dosage, which is candidate information to be reviewed by clinicians.

4. The method for processing multimodal data of TCM syndrome differentiation based on inference chain distillation according to claim 3, characterized in that: The gating function in step S2 also takes the normalized entropy of the output distribution of this stage as input; the structured dialectical reference information includes prompts for modal data to be supplemented, the modal data to be supplemented is the modal data corresponding to the modality with the lowest cumulative attention in this stage, and the cumulative attention is the sum of the cross-attention weights of each attention head and each decoder layer on the corresponding modal projection vector of the decoder in this stage; after the probability of the backtracking action in the first inference stage among the multiple inference stages is set to zero, the probabilities of the remaining actions are renormalized.

5. The method for processing multimodal data of TCM syndrome differentiation based on inference chain distillation according to claim 1, characterized in that: Step S3 updates the modality weight vector based on the cross-attention weights, including the following steps: adjusting the attention heads and decoder layers of the decoder in this stage to the... m The cross-attention weights of the modal projection vectors are summed to obtain the first... m The cumulative attention value of each modality is normalized by softmax with a temperature coefficient, and then used as the updated modality weight vector. The temperature coefficient is 0.2 to 1.0, and the upper limit is 1 to 3 times. When the increment of the maximum probability of the stage conclusion output after re-execution of inference in this stage is less than the confidence gain threshold, the process moves to the next stage. The stage conclusion probability distribution is the normalized distribution of the decoder output in this stage, and the confidence gain threshold is 0.01 to 0.

05.

6. The method for processing multimodal data of TCM syndrome differentiation based on inference chain distillation according to claim 2, characterized in that: The modal encoder includes: a visual Transformer encoder for extracting the modal feature vectors from the visual image data; a speech Transformer encoder for extracting the modal feature vectors from the auscultation speech data; a text Transformer encoder for extracting the modal feature vectors from the inquiry text data; and a one-dimensional residual convolutional network encoder for extracting the modal feature vectors from the palpation pulse data. The vector output by the one-dimensional residual convolutional network encoder after multiple layers of one-dimensional convolution and residual connections is concatenated with the heart rate variability features calculated from the palpation pulse data to obtain the modal feature vectors of the palpation pulse data. The cross-modal semantic alignment adopts a contrastive learning loss function, using different modal data of the same patient to form positive sample pairs and modal data of different patients to form negative samples, so that the modal projection vectors of the same patient are mutually aggregated in the shared diagnostic feature space.

7. The method for processing multimodal data of TCM syndrome differentiation based on inference chain distillation according to claim 3, characterized in that: The total loss of the inference chain distillation in step S4 is the weighted sum of the state distillation loss, the transition distillation loss, and the decision distillation loss according to the weight coefficients. The state distillation loss is as follows: the hidden states of the teacher model at each inference stage are projected to the same dimension space as the hidden states of the student model through a linear projection layer. The projected hidden state set of the teacher model and the hidden state set of the student model within the same training batch are used as two sample sets. The maximum mean difference between the two sample sets at each inference stage is measured by a Gaussian kernel and summed over each inference stage. The transfer distillation loss is the sum of the KL divergence between the transfer probability distributions of the teacher model and the student model at each inference stage at each inference stage; The decision distillation loss is the sum of the cross-entropy loss calculated by scaling the logarithmic probabilities of the proof type output by the teacher model and the student model using the distillation temperature, and the mean squared error loss that constrains the alignment of the gating weights used by the teacher model and the student model to generate the transition probability distribution at each inference stage.

8. The method for processing multimodal data of TCM syndrome differentiation based on inference chain distillation according to claim 7, characterized in that: The inference chain distillation is performed sequentially through four training phases using a progressive training strategy: The first training phase distills the reasoning chain knowledge from the first reasoning stage to the second reasoning stage; The second training phase adds distillation for the third inference phase on top of freezing the parameters trained in the first training phase. The third training phase adds distillation for the fourth inference phase based on freezing the parameters trained in the first and second training phases; The fourth training phase adds distillation for the fifth inference phase based on freezing the trained parameters from the first, second, and third training phases. The total loss for each training phase is a weighted sum of the state distillation loss, the transition distillation loss, and the decision distillation loss corresponding to the activated inference phase.

9. The method for processing multimodal data of TCM syndrome differentiation based on inference chain distillation according to claim 3, characterized in that: The structured dialectical reference information in step S4 includes: the evidence type determination result and the confidence probability distribution corresponding to the evidence type determination result; the transition probability distribution of each reasoning stage; and the actual transition path of the reasoning chain, wherein the actual transition path includes the sequence number of the reasoning stage in which the backtracking action occurs, the value of the modal weight vector before and after the backtracking action, and the confidence probability distribution before and after the backtracking action.

10. A multimodal data processing system for TCM syndrome differentiation based on inference chain distillation, characterized in that, The system includes: Multimodal data acquisition module: used to acquire pre-collected and stored four diagnostic modal data, including visual diagnosis image data, auscultation and olfaction voice data, inquiry text data, and palpation pulse data; Modality encoding and alignment module: used to encode the four diagnostic modal data into modality projection vectors, and to obtain a shared diagnostic feature vector by weighting and summing the modality projection vectors according to the modality weight vector; The chain-based reasoning module includes a teacher model and a student model, each with multiple reasoning stages. The decoder at each stage generates the hidden state of the current stage from the hidden state of the previous stage and the shared dialectical feature vector. Based on the hidden state of the current stage and the cross-attention weights of the decoder's modal projection vector, a gating function is used to calculate the transition probability distribution for forward, backward, and termination actions in the current stage. When the probability of the termination action is maximum, the chain-based reasoning module stops reasoning and outputs structured dialectical reference information for the completed reasoning stage and a prompt for supplementary modal data. When the probability of the backward action is greater than the probability of the forward action and the number of backward actions has not reached the upper limit, the chain-based reasoning module sends the cross-attention weights of the current stage to the modal encoding and alignment module. The modal encoding and alignment module updates the modal weight vector and recalculates the shared dialectical feature vector. The chain-based reasoning module then re-executes reasoning from the previous stage using the recalculated shared dialectical feature vector; otherwise, it moves to the next stage. Inference chain distillation module: used to align the transition probability distributions of the teacher model and the student model with KL divergence to complete inference chain distillation, and the student model outputs structured dialectical reference information.

Images