A traditional Chinese medicine four-examination auxiliary diagnosis and treatment system and method based on multi-modal feature fusion

Abstract

This invention provides a TCM four diagnostic methods auxiliary diagnosis and treatment system and method based on multimodal feature fusion, belonging to the field of intelligent auxiliary diagnosis and treatment technology. The system includes: acquiring and preprocessing raw TCM four diagnostic data to generate TCM four diagnostic data; performing feature extraction, vector transformation, and cross-modal fusion calculation on the TCM four diagnostic data to generate multimodal fusion feature vectors; using vector coding technology to vector-encode a medical knowledge inheritance experience knowledge base to generate a medical knowledge embedding matrix; based on the medical knowledge embedding matrix, using an optimized dialectical reasoning model to perform deep feature extraction and multi-task classification prediction on the multimodal fusion feature vectors to generate dialectical conclusion data; matching and combining recommended prescription schemes and specific care plans to summarize and generate TCM four diagnostic methods auxiliary diagnosis and treatment plans, thereby achieving collaborative analysis of TCM four diagnostic data and deep integration of medical inheritance experience, improving the accuracy, completeness, and decision-making efficiency of TCM auxiliary diagnosis and treatment.

Description

Technical Field

[0001] This invention relates to the field of intelligent auxiliary diagnosis and treatment technology, and in particular to a TCM four diagnostic methods auxiliary diagnosis and treatment system and method based on multimodal feature fusion. Background Technology

[0002] Traditional Chinese medicine (TCM) diagnosis and treatment rely on a comprehensive analytical approach that combines observation, auscultation and olfaction, inquiry, and palpation. Observation involves examining the tongue and complexion; auscultation involves listening to the voice and breath; inquiry involves taking notes on symptoms and medical history; and palpation involves feeling the pulse and pulsation. By comprehensively gathering physiological and pathological information about the patient, a TCM-assisted diagnostic and treatment plan based on syndrome differentiation is generated. This multi-dimensional and holistic TCM diagnostic and treatment model has been validated through thousands of years of clinical practice and possesses unique theoretical value and application advantages.

[0003] Current TCM-assisted diagnosis and treatment protocols primarily rely on physicians' accumulated clinical experience and subjective judgment. However, the training period for highly skilled TCM physicians is long, the resources for inheriting renowned veteran TCM doctors are limited, and the TCM diagnosis and treatment levels in primary healthcare institutions vary widely, making it difficult to meet the public's demand for high-quality TCM services. Furthermore, existing technologies largely focus on the digital processing of single-modality diagnostic data, failing to achieve collaborative integration and analysis of multimodal diagnostic data. The main problems with existing technologies are as follows: First, the lack of standardized processing and feature extraction mechanisms makes it difficult to effectively correlate and analyze diagnostic data from different modalities and to perform collaborative reasoning. Second, existing technologies mainly rely on general medical knowledge, failing to deeply integrate the inherited experience of medical schools, a large number of classic medical cases, and experience in modifying prescriptions, resulting in a low degree of matching between diagnostic conclusions and clinical practice. Third, the output of existing technologies is mostly a single diagnostic conclusion or syndrome classification, lacking a complete support system from diagnostic conclusion data to recommended prescriptions and specific care plans, making it difficult to effectively assist physicians in clinical decision-making.

[0004] Therefore, it is necessary to provide a TCM diagnostic system and method based on multimodal feature fusion to solve the above-mentioned technical problems. Summary of the Invention

[0005] To address the aforementioned technical problems, this invention provides a TCM four diagnostic methods auxiliary diagnosis and treatment system and method based on multimodal feature fusion, which solves the problems of difficulty in coordinating multimodal diagnostic data, insufficient embedding of medical school inheritance experience, single dimension of syndrome differentiation data, and weak completeness and low accuracy of generated treatment plans.

[0006] This invention provides a TCM four-diagnostic auxiliary system based on multimodal feature fusion, the system comprising: The data acquisition and preprocessing module is used to acquire the patient's original data of the four diagnostic methods of traditional Chinese medicine and preprocess it to generate data of the four diagnostic methods of traditional Chinese medicine. The data of the four diagnostic methods of traditional Chinese medicine includes tongue image data of inspection, pulse waveform data of palpation, audio signal data of auscultation, and text data of symptoms of inquiry. The feature extraction and fusion module is used to extract features and transform vectors from the visual tongue image data, palpation pulse waveform data, auscultation audio signal data, and inquiry symptom text data using image processing technology, wavelet packet decomposition algorithm, Mel-frequency cepstral coefficient feature extraction algorithm, and semantic coding technology, respectively. It also performs cross-modal fusion calculation through an attention-weighted fusion mechanism to generate a multimodal fusion feature vector. The knowledge base retrieval and encoding module is used to retrieve the medical heritage experience knowledge base and to use vector encoding technology to perform vector encoding on the medical heritage experience knowledge base to generate a medical knowledge embedding matrix. The model building and dialectical reasoning module is used to perform deep feature extraction and multi-task classification prediction on the multimodal fusion feature vector based on the medical knowledge embedding matrix and an optimized dialectical reasoning model to generate dialectical conclusion data. The scheme matching and interactive correction module is used to query the prescription knowledge base and the nursing rule base, match the recommended prescription scheme and specific nursing scheme corresponding to the syndrome differentiation conclusion data based on the index mapping relationship table, summarize and encapsulate the syndrome differentiation conclusion data, the recommended prescription scheme and the specific nursing scheme, generate a TCM four diagnostic auxiliary treatment scheme and push it to the interactive platform for physicians to refer to and correct.

[0007] Preferably, the process of acquiring and preprocessing the patient's original TCM four diagnostic methods data to generate TCM four diagnostic methods data includes tongue image data from inspection, pulse waveform data from palpation, audio signal data from auscultation, and text data of symptoms from inquiry, specifically including: The ring-shaped standardized light source component is dynamically adjusted according to the patient's posture, and the patient's oral cavity image data is acquired through a high-definition camera component. The edge detection algorithm is used to detect the edge contour of the patient's tongue, and the internal area of ​​the edge contour of the patient's tongue is preserved. Then, the light correction algorithm is used for illumination correction and the median filtering algorithm is used for noise removal to generate the tongue image data for visual diagnosis. A pressure sensor array is linearly arranged against the radial artery of the patient's wrist to collect pulse pulsation signals and transmit them to a signal conditioning circuit. The pulse pulsation signals are then amplified, noise filtered, and converted to digital values ​​by a signal amplifier, a low-pass filter, and an analog-to-digital converter in the signal conditioning circuit to generate the palpation pulse waveform data. The patient is guided to read and breathe deeply according to the preset standard reading content through the preset display unit. The reading voice signal and breathing audio signal are collected by the cardioid pickup unit in the directional microphone assembly and transmitted to the audio processing circuit. The reading voice signal and breathing audio signal are amplified, high-frequency component filtered out and analog-to-digital converted in sequence through the preamplifier, anti-aliasing filter and analog-to-digital converter in the audio processing circuit, and the data is summarized to generate the auscultation audio signal data. The patient's symptom characters are collected using a structured symptom questionnaire component, and the data is cleaned using regular expressions to retain valid patient symptom characters and generate the consultation symptom text data.

[0008] Preferably, the step of using image processing technology, wavelet packet decomposition algorithm, Mel-frequency cepstral coefficient feature extraction algorithm, and semantic coding technology to perform feature extraction and vector transformation on the visual tongue image data, palpation pulse waveform data, auscultation audio signal data, and inquiry symptom text data, specifically includes: The RGB color space of the tongue image data is split using channel separation technology to obtain the R-channel pixel grayscale matrix, G-channel pixel grayscale matrix, and B-channel pixel grayscale matrix. Mathematical statistical calculations are then performed to obtain the mean, variance, and skewness of the R-channel, G-channel, and B-channel, and the color distribution characteristics are summarized to generate the color distribution features. The gray-level co-occurrence matrix algorithm is used to calculate the gray-level co-occurrence relationship corresponding to the R-channel pixel gray-level matrix, the G-channel pixel gray-level matrix, and the B-channel pixel gray-level matrix, respectively. The texture roughness, texture uniformity, and texture directionality in the corresponding R-channel gray-level co-occurrence matrix, G-channel gray-level co-occurrence matrix, and B-channel gray-level co-occurrence matrix are generated and extracted, and the texture structure features are summarized to generate texture structure features. Using geometric morphology quantization technology, the area, perimeter, aspect ratio, and edge curvature changes of the tongue image data are calculated and summarized to generate geometric morphological features; The color distribution features, texture structure features and geometric morphology features are sequentially concatenated, and dimensionality reduction optimization is performed using principal component analysis algorithm to generate a tongue image feature vector for visual diagnosis. For the pulse waveform data, a high-pass filtering algorithm is used to perform baseline drift correction, and an adaptive thresholding algorithm is used to detect the position of the main peak and perform waveform period segmentation to obtain multiple single-period pulse waveform data. Based on the time-domain feature extraction algorithm, features are extracted from the multiple single-period pulse waveform data to obtain the amplitude value of the main peak. The slope of the rising segment of the main wave The slope of the main wave's descending segment Ratio of tidal peak amplitude to main wave peak amplitude The ratio of the amplitude of the diphthral peak to the amplitude of the main peak and the duration of single-cycle pulse waveform data The corresponding calculation formula is as follows: In the formula, Indicates the peak amplitude of the tidal wave; This indicates the amplitude of the diphtheria peak; This represents the amplitude value of the single-cycle pulse waveform data at time t; These represent the start and end times of a single-cycle pulse waveform data, respectively. This indicates the time t corresponding to the amplitude value of the main wave peak; This represents the amplitude value of the single-cycle pulse waveform data at time t corresponding to the i-th sampling point; This represents the amplitude value of the single-cycle palpation pulse waveform data at time t corresponding to the (i-1)th sampling point; Indicates the time interval between adjacent sampling points; Calculate and integrate the average amplitude of the main peak of the palpation pulse waveform data from multiple single cycles. Mean slope of the rising segment of the main wave Mean slope of the descending segment of the main wave Mean ratio of tidal peak amplitude to main wave peak amplitude Mean ratio of the amplitude of the diphthral peak to the amplitude of the main peak The average duration of single-cycle pulse waveform data Generate temporal features of pulse diagnosis ; The wavelet packet decomposition algorithm is used to perform multi-scale time-frequency decomposition on multiple single-cycle palpation pulse waveform data to obtain palpation pulse component signals. And extract the palpation pulse component signal. Corresponding energy percentage and clock frequency parameters Calculate and integrate the average energy percentage Average value of main frequency parameters Generate frequency domain features of palpation pulse. The corresponding calculation formula is as follows: In the formula, j represents the number of decomposition layers; k represents the frequency band number; This represents the decomposition of low-pass filter coefficients; This represents the decomposition of the high-pass filter coefficients; n represents the coefficient index; Indicates Fast Fourier Transform; This indicates the value of the independent variable corresponding to the maximum value; The time-domain features of the palpation pulse The frequency domain features of the palpation pulse The components are sequentially concatenated and then optimized for dimensionality reduction using the principal component analysis algorithm to generate a pulse diagnosis feature vector. The auscultation audio signal data is segmented using a sliding window framing technique to obtain multiple auscultation audio signal frames. The multiple auscultation audio signal frames are then weighted using a Hamming window function to obtain an auscultation audio signal enhancement frame. The enhanced frames of the auscultation audio signal are converted into an auscultation audio spectrum sequence by the Fast Fourier Transform and classified according to the frequency interval differences to obtain the reading speech spectrum sequence and the breathing audio spectrum sequence. The reading speech spectrum sequence and the breathing audio spectrum sequence are input into the Mel filter bank, and weighted filtering, logarithmic operation and discrete cosine transform are performed based on the Mel cepstral coefficient feature extraction algorithm to obtain the static features and dynamic features of the reading speech, the energy distribution features of the breathing spectrum and the rhythmic features of the breathing cycle. The static features of the reading speech, the dynamic features of the reading speech, the energy distribution features of the breathing spectrum and the rhythmic features of the breathing cycle are concatenated in sequence, and the dimensionality reduction optimization is performed by the principal component analysis algorithm to generate the auscultation audio feature vector. The text data of the consultation symptoms is mapped to the semantic data of the consultation symptoms according to the text-semantic mapping rules. The semantic data of the consultation symptoms is segmented and aligned with entities based on the standard lexicon of symptom terms. Then, the feature vector of the consultation symptoms is generated by using Transformer encoding technology.

[0009] Preferably, the attention-weighted fusion mechanism is used to perform cross-modal fusion calculations on the tongue feature vector from visual examination, the pulse feature vector from palpation, the audio feature vector from auscultation, and the symptom feature vector from inquiry to generate the multimodal fusion feature vector, specifically including: The tongue appearance feature vector, pulse appearance feature vector, auscultation audio feature vector, and symptom inquiry feature vector are mapped to the hidden feature space through a fully connected mapping layer to achieve dimensionality unification, thereby generating the hidden feature vector of the tongue appearance. Pulse diagnosis hidden layer feature vector Auscultation audio hidden layer feature vector and the hidden feature vector of symptoms during medical consultation And summarized into hidden feature vectors of the four diagnostic methods of traditional Chinese medicine. The hidden feature vectors of the four diagnostic methods of traditional Chinese medicine are analyzed using a learnable weight matrix. The feature mapping is performed, and the corresponding calculation formula is as follows: In the formula, Represents the query weight matrix; Represents the key weight matrix; Represents the value weight matrix; Represents the global query vector matrix; Represents the global key vector matrix; Represents a global value vector matrix; These represent the query vectors for tongue appearance by observation, pulse appearance by palpation, audio by auscultation, and symptoms by inquiry, respectively. These represent the key vectors for tongue appearance in visual diagnosis, pulse appearance in palpation, audio in auscultation and olfaction, and symptom inquiry, respectively. These represent the value vectors for tongue appearance during inspection, pulse appearance during palpation, audio value vectors during auscultation and olfaction, and symptom value vectors during inquiry, respectively. Based on the global query vector matrix The global key vector matrix The global value vector matrix The hidden feature vectors of the four diagnostic methods in Traditional Chinese Medicine are calculated based on the dot product similarity theory. The corresponding attention weight matrix and the attention weight matrix Perform matrix multiplication operations with different global value vectors respectively, and then summarize to obtain the enhanced feature representation. The enhanced feature representation is obtained by using a weighted fusion formula. Perform a weighted summation to generate the multimodal fusion feature vector. The corresponding calculation formula is as follows: In the formula, This represents the normalized exponential function; Represents the dimension of the global key vector matrix; These respectively represent the enhanced features of tongue appearance in visual diagnosis, enhanced features of pulse in palpation, enhanced audio in auscultation, and enhanced features of symptoms in inquiry. These represent the enhanced feature weights for visual diagnosis of the tongue, palpation of the pulse, auscultation of the audio, and inquiry of symptoms, respectively.

[0010] Preferably, the step of calling upon a medical heritage experience knowledge base and using vector encoding technology for vector encoding and concatenation to generate a medical knowledge embedding matrix specifically includes: The medical heritage knowledge base includes records of classic medical cases and summaries of diagnostic rules. Named entity recognition technology is used to extract entities from the classic medical records and label the entity categories. Candidate classic medical records are generated by summarizing them. Based on the standard symptom terminology lexicon, the candidate classic medical records are aligned to generate standard classic medical records. The summary of dialectical rules is logically analyzed using rule parsing technology to extract rule logical expressions. Based on the rule logical expressions, the summary of dialectical rules is structurally reconstructed to generate a standard summary of dialectical rules. The vector encoding technology is used to perform vector encoding on the standard classic medical case records and the standard dialectical rule summary, respectively, to generate feature vectors of classic medical case records and feature vectors of dialectical rule summary, which are then horizontally concatenated to generate the medical knowledge embedding matrix.

[0011] Preferably, the step of using an optimized dialectical reasoning model to perform deep feature extraction and multi-task classification prediction on the multimodal fusion feature vector based on the medical knowledge embedding matrix to generate dialectical conclusion data specifically includes: The optimized dialectical reasoning model, which cascades an encoder and decoder, integrates multimodal feature vectors. Medical knowledge embedding matrix Input to encoder and calculate cosine similarity The corresponding calculation formula is as follows: In the formula, Represents L2 norm operations; A preset cosine similarity threshold is set, when the cosine similarity... If the cosine similarity is greater than or equal to the cosine similarity threshold, then the cosine similarity is filtered. The classic medical case record feature vector corresponding to the medical knowledge embedding matrix is ​​used, and the selected classic medical case record feature vector is horizontally concatenated with the multimodal fusion feature vector to generate a knowledge-enhanced feature vector; The fully connected classification head network in the decoder is used to perform parallel inference on the knowledge-enhanced feature vector to output the probability distribution of constitution type, syndrome type, and risk level. Combined with the maximum value sampling strategy, the corresponding constitution type prediction result, syndrome type prediction result, and risk level prediction result are output. The constitution type prediction result, syndrome type prediction result, and risk level prediction result are integrated and encapsulated to generate the dialectical conclusion data.

[0012] Preferably, the query of the prescription knowledge base and the care rule base, based on the index mapping relationship table, matches the recommended prescription scheme and specific care scheme corresponding to the syndrome differentiation conclusion data, specifically including: Based on the diagnostic conclusion data, the formula knowledge base is queried, and the syndrome type prediction result is used as the index to search in the syndrome-formula mapping table to obtain multiple candidate formula schemes; Based on the syndrome type prediction results and the candidate prescription schemes, the cosine similarity algorithm is used to calculate the prescription matching score. Based on the predicted constitution type and the candidate prescriptions, the cosine similarity algorithm is used to calculate the constitution matching score. ; The formula will be matched and scored. Matching score with the physical condition The weighted fusion is used to obtain the comprehensive score of the prescription. In the formula, This indicates the weight of the prescription matching score; The weight of the constitution matching score is indicated by the highest comprehensive score of the prescribed formula. The corresponding candidate prescription scheme is used as the first prescription scheme; The formula addition and subtraction rule library is retrieved, and the tongue appearance feature vector, pulse appearance feature vector, auscultation and olfaction audio feature vector, and symptom feature vector are input into the formula addition and subtraction condition expression for logical judgment. If the output logical judgment result is true, the feature vector-formula addition and subtraction mapping table is triggered to perform the corresponding formula addition and subtraction operation and update the first formula scheme to generate the second formula scheme. The second prescription scheme is input into the drug compatibility data table and a group of prescriptions in the second prescription scheme is randomly traversed. If the group of prescriptions exists in the drug compatibility data table, a compatibility warning message and a prescription replacement scheme are generated. The compatibility warning information, the prescription replacement scheme, and the set of prescriptions are returned to the prescription addition and subtraction rule library, and prescription addition and subtraction operations are performed on the set of prescriptions according to the prescription replacement scheme to obtain the recommended prescription scheme; The syndrome type prediction results and the constitution type prediction results are input into the nursing care rule base, and the specific nursing care plan is obtained based on the syndrome-nursing care mapping table and the constitution-nursing care mapping table.

[0013] Preferably, after pushing the TCM four diagnostic methods auxiliary treatment plan to the interactive platform for physicians to refer to and revise, it further includes: Physicians can view the TCM four diagnostic methods auxiliary treatment plan through the visual interface of the interactive platform and make manual corrections to generate the final TCM four diagnostic methods auxiliary treatment plan; The correction content, correction tags, and the final TCM four diagnostic methods auxiliary treatment plan recorded by the interactive platform are transmitted to the optimized syndrome differentiation reasoning model, and an incremental learning algorithm is used for self-updating and optimization.

[0014] A TCM diagnostic and treatment method based on multimodal feature fusion, comprising: The patient's original data from the four diagnostic methods of traditional Chinese medicine (TCM) are acquired and preprocessed to generate TCM diagnostic data, which includes tongue image data from inspection, pulse waveform data from palpation, audio signal data from auscultation, and text data of symptoms from inquiry. Image processing technology, wavelet packet decomposition algorithm, Mel-frequency cepstral coefficient feature extraction algorithm, and semantic coding technology are used to extract features and transform vectors from the visual tongue image data, palpation pulse waveform data, auscultation audio signal data, and inquiry symptom text data, respectively. Cross-modal fusion calculation is performed through attention-weighted fusion mechanism to generate multimodal fusion feature vectors. The medical heritage experience knowledge base is invoked, and vector encoding technology is used to vector encode the medical heritage experience knowledge base to generate a medical knowledge embedding matrix; Based on the medical knowledge embedding matrix, an optimized dialectical reasoning model is used to perform deep feature extraction and multi-task classification prediction on the multimodal fusion feature vector to generate dialectical conclusion data. The system queries the formula knowledge base and nursing rule base, matches the recommended formula schemes and specific nursing schemes corresponding to the syndrome differentiation data based on the index mapping relationship table, summarizes and encapsulates the syndrome differentiation data, the recommended formula schemes and the specific nursing schemes, generates a TCM four diagnostic and treatment auxiliary treatment plan, and pushes it to the interactive platform for physicians to refer to and correct.

[0015] Compared with related technologies, the TCM four diagnostic methods auxiliary diagnosis and treatment system and method based on multimodal feature fusion provided by the present invention have the following beneficial effects: This invention employs a data acquisition and preprocessing module to acquire and preprocess raw data from patients using the four diagnostic methods of Traditional Chinese Medicine (TCM), generating TCM diagnostic data. This data includes tongue image data (observation), pulse waveform data (palpation), audio signal data (auscultation), and textual data of symptoms (inquiry). A feature extraction and fusion module utilizes image processing techniques, wavelet packet decomposition algorithms, Mel-frequency cepstral coefficient feature extraction algorithms, and semantic coding techniques to extract features and transform vectors from the tongue image data, pulse waveform data, audio signal data, and textual data of symptoms, respectively. An attention-weighted fusion mechanism is then used to perform cross-modal fusion calculations, generating a multimodal fusion feature vector. Finally, a knowledge base retrieval and encoding module is used. This system is used to access a medical heritage experience knowledge base and employs vector encoding technology to generate a medical knowledge embedding matrix. The model building and dialectical reasoning module, based on the medical knowledge embedding matrix, uses an optimized dialectical reasoning model to perform deep feature extraction and multi-task classification prediction on multimodal fusion feature vectors, generating dialectical conclusion data. The scheme matching and interactive correction module queries the prescription knowledge base and nursing rule base, matches recommended prescription schemes and specific nursing schemes corresponding to the dialectical conclusion data based on an index mapping table, summarizes and encapsulates the dialectical conclusion data, recommended prescription schemes, and specific nursing schemes, generates a TCM four-diagnosis auxiliary treatment plan, and pushes it to the interactive platform for physicians to refer to and correct.

[0016] This invention achieves unified representation and deep fusion of tongue image data from visual examination, pulse waveform data from palpation, audio signal data from auscultation, and textual data from symptom inquiry by constructing a standardized processing and feature extraction mechanism for multimodal diagnostic data. This overcomes the limitations of existing technologies that process data independently from a single diagnostic method, effectively improving the accuracy and completeness of collaborative analysis of multimodal diagnostic data. Furthermore, by deeply embedding the inherited experience of medical schools into an optimized diagnostic reasoning model using vector coding technology, it enables dynamic retrieval and knowledge enhancement of valuable clinical knowledge such as classic medical cases and diagnostic rules, significantly improving the clinical matching degree and theoretical solvability of diagnostic conclusion data. The system is characterized by its comprehensive nature. By establishing a full-chain solution generation mechanism encompassing "syndrome differentiation, prescription, and nursing care," it achieves a complete output of TCM four-diagnosis auxiliary treatment plans, from constitution prediction, syndrome judgment, risk prediction, prescription matching, personalized addition and subtraction to nursing care suggestions. Furthermore, it incorporates a drug compatibility data table to detect and ensure medication safety, significantly enhancing the systematicness, personalization, and practicality of TCM four-diagnosis auxiliary treatment plans. Through the construction of a human-machine collaborative interactive correction and incremental learning mechanism, it achieves a closed-loop iteration of physician correction feedback and optimization of the syndrome differentiation reasoning model, enabling the system to continuously learn and self-evolve, effectively adapting to the dynamic evolution of clinical needs. Attached Figure Description

[0017] Figure 1 This is a system block diagram of a TCM four-diagnosis auxiliary diagnosis and treatment system based on multimodal feature fusion provided in an embodiment of the present invention; Figure 2 The flowchart illustrates a TCM diagnostic and treatment method based on multimodal feature fusion, which is provided as an embodiment of the present invention. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the present invention will be further described below with reference to the accompanying drawings and embodiments. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention. Example 1

[0019] like Figure 1 The image shows a TCM diagnostic system based on multimodal feature fusion, provided by an embodiment of the present invention. The system includes: The data acquisition and preprocessing module is used to acquire the patient's original data of the four diagnostic methods of traditional Chinese medicine and preprocess it to generate data of the four diagnostic methods of traditional Chinese medicine. The data of the four diagnostic methods of traditional Chinese medicine includes tongue image data of inspection, pulse waveform data of palpation, audio signal data of auscultation, and text data of symptoms of inquiry. The data includes: visual tongue image data (observation), which refers to the patient's tongue image information acquired and preprocessed using a high-definition camera component, including visual features such as tongue color, tongue coating texture, and tongue shape; pulse waveform data (palpation), which refers to the radial artery pulse signal at the wrist acquired and preprocessed using a pressure sensor array, reflecting the temporal and frequency domain characteristics of the pulse; audio signal data (auscultation), which refers to the reading voice signal and breathing audio signal acquired and preprocessed using a directional microphone component, including acoustic features such as pitch and breath rate; and textual data (questioning), which refers to the patient's subjective symptoms, medical history, drug contraindications, and lifestyle habits acquired and preprocessed using a structured symptom questionnaire.

[0020] Understandably, the data acquisition and preprocessing module can achieve standardized acquisition, precise preprocessing, and unified representation of raw data from the four diagnostic methods of traditional Chinese medicine. It solves the technical problems of strong subjectivity, inconsistent formats, and large noise interference in existing technologies, and realizes unified quality control and standardized output of tongue image data from inspection, pulse waveform data from palpation, audio signal data from auscultation, and text data of symptoms from inquiry.

[0021] The process involves acquiring and preprocessing the patient's original data from the four diagnostic methods of Traditional Chinese Medicine (TCM) to generate TCM diagnostic data. This TCM diagnostic data includes tongue image data from inspection, pulse waveform data from palpation, audio signal data from auscultation, and textual data of symptoms from inquiry. Specifically, it includes: The ring-shaped standardized light source component is dynamically adjusted according to the patient's posture, and the patient's oral cavity image data is acquired through a high-definition camera component. The edge detection algorithm is used to detect the edge contour of the patient's tongue, and the internal area of ​​the edge contour of the patient's tongue is preserved. Then, the light correction algorithm is used for illumination correction and the median filtering algorithm is used for noise removal to generate the tongue image data for visual diagnosis. A pressure sensor array is linearly arranged against the radial artery of the patient's wrist to collect pulse pulsation signals and transmit them to a signal conditioning circuit. The pulse pulsation signals are then amplified, noise filtered, and converted to digital values ​​by a signal amplifier, a low-pass filter, and an analog-to-digital converter in the signal conditioning circuit to generate the palpation pulse waveform data. The patient is guided to read and breathe deeply according to the preset standard reading content through the preset display unit. The reading voice signal and breathing audio signal are collected by the cardioid pickup unit in the directional microphone assembly and transmitted to the audio processing circuit. The reading voice signal and breathing audio signal are amplified, high-frequency component filtered out and analog-to-digital converted in sequence through the preamplifier, anti-aliasing filter and analog-to-digital converter in the audio processing circuit, and the data is summarized to generate the auscultation audio signal data. The patient's symptom characters are collected using a structured symptom questionnaire component, and the data is cleaned using regular expressions to retain valid patient symptom characters and generate the consultation symptom text data.

[0022] The components include: a ring-shaped standardized light source assembly (which dynamically adjusts the angle, brightness, and uniformity of illumination based on the patient's head posture to eliminate interference from ambient light changes in the acquisition of oral cavity image data); an edge detection algorithm (which identifies the outline of the patient's tongue by calculating the color gradient values ​​of each pixel in the oral cavity image data to accurately segment the tongue region from the background region); a pressure sensor array (a linearly arranged sensor group composed of multiple miniature pressure sensing elements, each corresponding to one of the three traditional pulse diagnosis sites: cun, guan, and chi, used to simultaneously acquire pulse pressure change signals from multiple sites); a cardioid directional microphone assembly (which has high sensitivity to forward-facing speech and respiratory audio signals and strong suppression of lateral and backward ambient noise, used to acquire high-quality speech and respiratory audio signals from patients); and a structured symptom questionnaire assembly (an electronic questionnaire with pre-set standardized symptom options and open input fields, used to standardize the collection of textual information such as patients' subjective symptoms, medical history, drug contraindications, and lifestyle habits). Regular expressions are string matching rules used for data cleaning. They filter out invalid patient symptom characters such as special symbols, repetitive expressions, and ambiguous semantics through pattern recognition, and fill in missing patient symptom characters.

[0023] Specifically, in the process of acquiring tongue image data for visual diagnosis, firstly, the illumination angle and intensity of the ring-shaped standardized light source component are dynamically adjusted according to the patient's facial posture to ensure uniform illumination and no shadows on the patient's tongue surface. Then, a high-definition camera component is positioned 30 centimeters away from the patient's tongue, capturing images of the patient's oral cavity at a preset focal length and using an image resolution of at least 20 megapixels to clearly present the subtle textures and color variations of the tongue surface. In the stage of detecting the patient's tongue edge contour, an edge detection algorithm is used to calculate the color gradient value of each pixel in the patient's oral cavity image data. A dual-threshold algorithm is used to determine the patient's tongue edge contour, and morphological closing operations are applied to smooth the edges of the patient's tongue edge contour. After marking the internal area of ​​the patient's tongue edge contour as the tongue region, a gamma correction algorithm is used to compensate for nonlinear illumination deviations, and a median filtering algorithm is used to remove noise, ultimately generating standardized tongue image data for visual diagnosis.

[0024] Furthermore, during the acquisition of palpation pulse waveform data, the pressure sensor array is placed against the radial artery of the patient's wrist and held still for 30 seconds. Each miniature pressure sensor element synchronously acquires pulse pulsation signals from the three traditional pulse diagnosis sites of cun, guan, and chi at a sampling frequency of ≥1000Hz, ensuring that the acquired pulse pulsation signals contain several complete pulse cycles and fully reflect the periodicity and stability of the palpation pulse waveform data. The signal amplifier in the signal conditioning circuit adopts an instrumentation amplifier structure to amplify the weak pulse pulsation signal to a preset amplitude range. Then, a low-pass filter is used to effectively filter out power frequency interference and high-frequency noise. Subsequently, an analog-to-digital converter is used to convert the analog pulse pulsation signal into a digital pulse pulsation signal, thereby ensuring the dynamic range of the generated palpation pulse waveform data.

[0025] Then, during the acquisition of auscultation audio signal data, the preset display unit refers to a 10-15 inch LCD interactive display screen, used to display preset standard reading content and deep breathing guidance instructions, guiding the patient to read aloud for 10 seconds at a natural speaking speed and perform standardized deep breathing. The cardioid pickup unit continuously records the reading voice signal during the patient's vocalization, with high pickup sensitivity to sound sources directly in front and strong suppression of lateral and rearward environmental noise. After the patient finishes reading, the preset display unit guides the patient to perform deep breathing actions, and the cardioid pickup unit simultaneously records the respiratory audio signals during inhalation and exhalation. The preamplifier in the audio processing circuit amplifies the weak respiratory audio signal to a standard level range, the anti-aliasing filter filters out high-frequency components above 8kHz to avoid spectral aliasing, and the analog-to-digital converter converts the analog reading voice signal and analog respiratory audio signal into digital reading voice signal and digital respiratory audio signal, respectively, and summarizes them to generate standardized auscultation audio signal data.

[0026] Finally, during the collection of textual data on symptoms during the consultation, the structured symptom questionnaire component presents preset standardized symptom options through a touch-screen interface. These options include subjective symptom dimensions such as chills and fever, sweating, diet, sleep, bowel movements, emotions, and the location and nature of pain, as well as auxiliary information fields such as medical history, drug allergy history, and lifestyle habits. After the patient selects a preset standardized symptom option or enters an open field, the system uses regular expressions for real-time data cleaning. Through pattern recognition, invalid patient symptom characters such as special symbols, repetitive expressions, and ambiguous semantics are filtered out, and missing patient symptom characters are filled in to generate structured textual data on symptoms during the consultation.

[0027] The feature extraction and fusion module is used to extract features and transform vectors from the visual tongue image data, palpation pulse waveform data, auscultation audio signal data, and inquiry symptom text data using image processing technology, wavelet packet decomposition algorithm, Mel-frequency cepstral coefficient feature extraction algorithm, and semantic coding technology, respectively. It also performs cross-modal fusion calculation through an attention-weighted fusion mechanism to generate a multimodal fusion feature vector. It should be noted that the feature extraction and fusion module is used to achieve standardized feature extraction and cross-modal deep fusion of tongue image data from visual examination, pulse waveform data from palpation, audio signal data from auscultation, and textual data from symptom consultation. Image processing techniques are used to extract three types of feature information—color, texture, and morphology—from the tongue image data. Wavelet packet decomposition is used to perform multi-scale time-frequency decomposition on the pulse waveform data, achieving comprehensive analysis from the time domain to the frequency domain. Mel-frequency cepstral coefficient feature extraction is used to extract acoustic feature information corresponding to the audio signal data from auscultation that conforms to the characteristics of human hearing. Semantic coding techniques are used to eliminate the diversity and ambiguity of linguistic expression in the textual data of symptom consultation, achieving standardized representation of the textual data.

[0028] The process employs image processing techniques, wavelet packet decomposition algorithms, Mel-frequency cepstral coefficient feature extraction algorithms, and semantic coding techniques to perform feature extraction and vector transformation on the visual examination tongue image data, the palpation pulse waveform data, the auscultation audio signal data, and the inquiry symptom text data, specifically including: The RGB color space of the tongue image data is split using channel separation technology to obtain the R-channel pixel grayscale matrix, G-channel pixel grayscale matrix, and B-channel pixel grayscale matrix. Mathematical statistical calculations are then performed to obtain the mean, variance, and skewness of the R-channel, G-channel, and B-channel, and the color distribution characteristics are summarized to generate the color distribution features. The gray-level co-occurrence matrix algorithm is used to calculate the gray-level co-occurrence relationship corresponding to the R-channel pixel gray-level matrix, the G-channel pixel gray-level matrix, and the B-channel pixel gray-level matrix, respectively. The texture roughness, texture uniformity, and texture directionality in the corresponding R-channel gray-level co-occurrence matrix, G-channel gray-level co-occurrence matrix, and B-channel gray-level co-occurrence matrix are generated and extracted, and the texture structure features are summarized to generate texture structure features. Using geometric morphology quantization technology, the area, perimeter, aspect ratio, and edge curvature changes of the tongue image data are calculated and summarized to generate geometric morphological features; The color distribution features, texture structure features and geometric morphology features are sequentially concatenated, and dimensionality reduction optimization is performed using principal component analysis algorithm to generate a tongue image feature vector for visual diagnosis. For the pulse waveform data, a high-pass filtering algorithm is used to perform baseline drift correction, and an adaptive thresholding algorithm is used to detect the position of the main peak and perform waveform period segmentation to obtain multiple single-period pulse waveform data. Based on the time-domain feature extraction algorithm, features are extracted from the multiple single-period pulse waveform data to obtain the amplitude value of the main peak. The slope of the rising segment of the main wave The slope of the main wave's descending segment Ratio of tidal peak amplitude to main wave peak amplitude The ratio of the amplitude of the diphthral peak to the amplitude of the main peak and the duration of single-cycle pulse waveform data The corresponding calculation formula is as follows: In the formula, Indicates the peak amplitude of the tidal wave; This indicates the amplitude of the diphtheria peak; This represents the amplitude value of the single-cycle pulse waveform data at time t; These represent the start and end times of a single-cycle pulse waveform data, respectively. This indicates the time t corresponding to the amplitude value of the main wave peak; This represents the amplitude value of the single-cycle pulse waveform data at time t corresponding to the i-th sampling point; This represents the amplitude value of the single-cycle palpation pulse waveform data at time t corresponding to the (i-1)th sampling point; Indicates the time interval between adjacent sampling points; Calculate and integrate the average amplitude of the main peak of the palpation pulse waveform data from multiple single cycles. Mean slope of the rising segment of the main wave Mean slope of the descending segment of the main wave Mean ratio of tidal peak amplitude to main wave peak amplitude Mean ratio of the amplitude of the diphthral peak to the amplitude of the main peak The average duration of single-cycle pulse waveform data Generate temporal features of pulse diagnosis ; The wavelet packet decomposition algorithm is used to perform multi-scale time-frequency decomposition on multiple single-cycle palpation pulse waveform data to obtain palpation pulse component signals. And extract the palpation pulse component signal. Corresponding energy percentage and clock frequency parameters Calculate and integrate the average energy percentage Average value of main frequency parameters Generate frequency domain features of palpation pulse. The corresponding calculation formula is as follows: In the formula, j represents the number of decomposition layers; k represents the frequency band number; This represents the decomposition of low-pass filter coefficients; This represents the decomposition of the high-pass filter coefficients; n represents the coefficient index; Indicates Fast Fourier Transform; This indicates the value of the independent variable corresponding to the maximum value; The time-domain features of the palpation pulse The frequency domain features of the palpation pulse The components are sequentially concatenated and then optimized for dimensionality reduction using the principal component analysis algorithm to generate a pulse diagnosis feature vector. The auscultation audio signal data is segmented using a sliding window framing technique to obtain multiple auscultation audio signal frames. The multiple auscultation audio signal frames are then weighted using a Hamming window function to obtain an auscultation audio signal enhancement frame. The enhanced frames of the auscultation audio signal are converted into an auscultation audio spectrum sequence by the Fast Fourier Transform and classified according to the frequency interval differences to obtain the reading speech spectrum sequence and the breathing audio spectrum sequence. The reading speech spectrum sequence and the breathing audio spectrum sequence are input into the Mel filter bank, and weighted filtering, logarithmic operation and discrete cosine transform are performed based on the Mel cepstral coefficient feature extraction algorithm to obtain the static features and dynamic features of the reading speech, the energy distribution features of the breathing spectrum and the rhythmic features of the breathing cycle. The static features of the reading speech, the dynamic features of the reading speech, the energy distribution features of the breathing spectrum and the rhythmic features of the breathing cycle are concatenated in sequence, and the dimensionality reduction optimization is performed by the principal component analysis algorithm to generate the auscultation audio feature vector. The text data of the consultation symptoms is mapped to the semantic data of the consultation symptoms according to the text-semantic mapping rules. The semantic data of the consultation symptoms is segmented and aligned with entities based on the standard lexicon of symptom terms. Then, the feature vector of the consultation symptoms is generated by using Transformer encoding technology.

[0029] Among them, the dimension of the tongue feature vector for visual diagnosis is 128, the dimension of the pulse feature vector for palpation is 64, the dimension of the audio feature vector for auscultation is 48, and the dimension of the symptom feature vector for inquiry is 32.

[0030] First, channel separation technology is used to split the RGB color space containing the tongue image data for visual diagnosis into independent R-channel pixel grayscale matrices, G-channel pixel grayscale matrices, and B-channel pixel grayscale moments. Then, through mathematical statistical calculations on the R-channel pixel grayscale matrices, G-channel pixel grayscale matrices, and B-channel pixel grayscale moments, the mean, variance, and skewness of the R-channel, G-channel, and B-channel are obtained. Among them, the mean is used to reflect the color depth of the tongue body in each channel; the variance is used to reflect the dispersion of the tongue body color distribution in each channel; and the skewness is used to reflect the asymmetry of the tongue body color distribution in each channel, thereby accurately capturing the color distribution characteristics of the tongue body, such as red-purple and yellow-white tongue coating. The mean, variance, and skewness of the R channel reflect the redness of the tongue body, and can distinguish different tongue body states such as pale white tongue, red tongue, and crimson tongue; the mean, variance, and skewness of the G channel and the mean, variance, and skewness of the B channel reflect the color changes of the tongue coating, such as white, yellow, gray, and black, thus breaking the coupling of the RGB color space and accurately distinguishing the color distribution characteristics of the tongue body and tongue coating.

[0031] Then, using the gray-level co-occurrence matrix algorithm, the pixel distance is set to 1-3 pixel units, and the orientation angle is set to 0°, 45°, 90°, and 135°. The corresponding R-channel gray-level co-occurrence matrices, G-channel gray-level matrices, and B-channel gray-level matrices are calculated respectively. Based on the gray-level co-occurrence matrices of each channel, texture roughness, texture uniformity, and texture orientation are extracted. Texture roughness reflects the coarseness and ridges of the tongue coating surface, corresponding to variations in coating thickness, thus distinguishing between thin and thick coatings. Thick tongue coatings correspond to higher texture roughness, while thin tongue coatings correspond to lower texture roughness. Texture uniformity reflects the evenness of tongue coating distribution, distinguishing between the integrity and peeling of the coating, thus identifying abnormalities such as peeling or geographic tongue. Evenly distributed tongue coating indicates higher texture uniformity, while partial peeling indicates lower uniformity. Texture directionality reflects the directional pattern of the tongue coating texture, corresponding to its fineness or roughness. Fine and uniform tongue coatings exhibit strong texture directionality, while rough and irregular tongue coatings show weaker directionality. Through collaborative texture analysis using R, G, and B channels, texture roughness, texture uniformity, and texture directionality are aggregated to generate texture structure features, thereby achieving precise quantitative differentiation of tongue coating thickness, integrity, and texture, avoiding the ambiguity and subjective differences inherent in manually defining qualitative descriptions such as thick, thin, complete, and peeling.

[0032] Finally, geometric morphology quantification technology is used to calculate geometric parameters such as area, perimeter, aspect ratio, and edge curvature changes in the tongue image data for visual diagnosis. These parameters reflect the thickness of the tongue and the presence of teeth marks, thus converting the tongue morphology into geometric parameters, objectively quantifying the tongue's morphological characteristics, and eliminating the differences in subjective human judgment. Color distribution features, texture structure features, and geometric morphology features are sequentially concatenated, and dimensionality reduction optimization is performed using principal component analysis to form a 128-dimensional tongue image feature vector, which is then stored in the tongue image channel of the feature vector cache.

[0033] Specifically, a high-pass filtering algorithm is used to correct baseline drift in the palpation pulse waveform data, removing low-frequency drift components caused by the patient's breathing or limb movements, thus restoring the baseline of the palpation pulse waveform data to a horizontal state. The palpation pulse waveform data exhibits a periodic pattern; a single-cycle palpation pulse waveform contains pulse information such as the main wave, tidal wave, and dicrotic wave, corresponding to pulse elements in traditional Chinese medicine such as superficial, deep, slow, rapid, slippery, and hesitant. Therefore, by using an adaptive threshold algorithm to detect the position of the main wave peak in the palpation pulse waveform data and segmenting the waveform period, continuous palpation pulse waveform data is divided into multiple independent single-cycle palpation pulse waveform data, covering all pulse information of a complete cycle. The amplitude of the main wave peak extracted using the time-domain feature extraction algorithm reflects the strength and depth of the pulse, used to determine the strength of the body's resistance and the depth of pathogenic factors. The slope of the main wave rising segment reflects the speed and smoothness of the pulse, used to determine the state of blood circulation and vascular vasomotor function. The slope of the main wave falling segment reflects the rate of pulse attenuation and vascular recoil characteristics, used to determine peripheral circulatory resistance and vascular elasticity. The ratio of the extracted tidal wave peak amplitude to the main wave peak amplitude reflects vascular tension and the tightness of the pulse, used to determine the degree of arteriosclerosis and changes in vascular compliance. The ratio of the extracted dicrotic wave peak amplitude to the main wave peak amplitude reflects vascular elasticity reserve and the gentleness of the pulse, used to determine vascular age and diastolic function reserve. The extracted single-cycle duration reflects the pulse rate and rhythm stability, used to determine cold / heat attributes and cardiac arrhythmias. By calculating and summarizing the average amplitude of the main wave peak, the average slope of the main wave rising segment, the average slope of the main wave falling segment, the average ratio of the tidal wave peak amplitude to the main wave peak amplitude, the average ratio of the dicrotic wave peak amplitude to the main wave peak amplitude, and the average duration of single-cycle palpation pulse waveform data, random interference can be eliminated, the patient's stable pulse state can be quantified, and stable and reliable palpation pulse time-domain characteristics can be generated.

[0034] By defining the decomposition level j and frequency band number k, and using iterative filtering based on the coefficients of the decomposition low-pass filter and high-pass filter, the single-cycle palpation pulse waveform data is decomposed into multiple frequency band palpation pulse component signals, fully covering the entire frequency range from the lowest to the highest frequency band. The energy value corresponding to each palpation pulse component signal is calculated through integration, and then the ratio of this energy value to the energy values ​​of all palpation pulse component signals is calculated to obtain the energy proportion of each palpation pulse component signal, reflecting the distribution pattern of energy values ​​in different frequency bands. The spectral distribution corresponding to each palpation pulse component signal is calculated through Fast Fourier Transform, and the dominant frequency parameter is extracted from the spectral distribution using a maximum value sampling strategy, reflecting the dominant frequency of pulse oscillation. The average energy percentage and the average main frequency parameter are summarized to generate stable and reliable frequency domain features of palpation pulse, which solves the problem of subjective judgment in traditional pulse diagnosis. The time domain features and frequency domain features of palpation pulse at the three traditional pulse diagnosis sites of cun, guan and chi are sequentially spliced ​​and optimized by dimensionality reduction to form a 64-dimensional palpation pulse feature vector and stored in the pulse channel of the feature vector cache area.

[0035] Then, the continuous auscultation audio signal data is segmented into multiple auscultation audio signal frames using a sliding window framing technique, with a frame length of 20-40 milliseconds and a frame shift of 10-20 milliseconds. Subsequently, the multiple auscultation audio signal frames are weighted using a Hamming window function to effectively suppress spectral leakage and edge effects in spectral analysis, enhancing the stability of the auscultation audio signal frames and obtaining enhanced auscultation audio signal frames. A Fast Fourier Transform is performed on the enhanced auscultation audio signal frames, converting them into auscultation audio spectrum sequences. Frequency classification is then performed based on frequency range differences, dividing the mid-to-high frequency auscultation audio spectrum sequences into reading speech spectrum sequences and the low-frequency auscultation audio spectrum sequences into breathing audio spectrum sequences, effectively separating the two signals and avoiding mutual interference.

[0036] Subsequently, the spectral sequences of the reading speech and breathing audio were input into a Mel filter bank composed of multiple filters. Based on the Mel cepstral coefficient feature extraction algorithm, the spectral sequences of the reading speech and breathing audio were nonlinearly weighted by weighted filtering to simulate the auditory characteristics of the human ear, specifically highlighting the acoustic features related to auscultation in traditional Chinese medicine. Logarithmic operations were then performed on the weighted filtered spectral sequences of the reading speech and breathing audio, effectively compressing the dynamic range of the two signals and reducing the interference of environmental noise. Discrete cosine transforms were then performed on the logarithmically processed spectral sequences of the reading speech and breathing audio, and orthogonal transformations were used to convert the correlation features of the spectral sequences of the reading speech and breathing audio into independent Mel cepstral coefficients, removing the linear correlation between the spectral sequences of the reading speech and breathing audio and reducing redundancy. For the spectral sequence of read-aloud speech after discrete cosine transform (DCT), statistical parameters such as the mean, variance, maximum, and minimum values ​​of the Mel-frequency cepstral coefficients are calculated and summarized to generate static features of the read-aloud speech, reflecting the stable acoustic properties of the spectral sequence. Simultaneously, the mean and variance of the first and second differences of the Mel-frequency cepstral coefficients are calculated and summarized to generate dynamic features of the read-aloud speech, capturing the changing trends of the spectral sequence. Based on a preset frequency range, the spectral sequence of breathing audio after DCT is divided into low-frequency and mid-to-high-frequency breathing audio sequences. The ratio of the energy of the low-frequency breathing audio sequence to the total energy of the breathing audio sequence, and the ratio of the energy of the mid-to-high-frequency breathing audio sequence to the total energy of the breathing audio sequence, are calculated and summarized to generate breathing spectrum energy distribution characteristics. A higher energy proportion in the low-frequency breathing audio sequence reflects deep and sufficient breathing, while a higher energy proportion in the mid-to-high-frequency breathing audio sequence reflects rapid and unstable breathing. The peak detection algorithm is used to detect the inspiratory and expiratory inflection points corresponding to the respiratory audio spectrum sequence after discrete cosine transform, and the inspiratory duration, expiratory duration, respiratory cycle, and inspiratory-expiratory ratio are calculated and summarized to obtain the respiratory cycle rhythm features. The static features of the reading speech, the dynamic features of the reading speech, the respiratory spectrum energy distribution features, and the respiratory cycle rhythm features are sequentially concatenated, and the dimensionality reduction optimization is performed by the principal component analysis algorithm to generate a 48-dimensional auscultation audio feature vector, which is then stored in the auscultation channel of the feature vector cache area.

[0037] Finally, based on preset text-semantic mapping rules, the text data of consultation symptoms is mapped into structured semantic data of consultation symptoms, eliminating the diversity and ambiguity of natural language expressions. The text data of consultation symptoms is identified and semantically normalized using text-semantic mapping rules, outputting the corresponding semantic data of consultation symptoms. Based on a standard symptom terminology lexicon, the semantic data of consultation symptoms is segmented and entity aligned, preserving symptom attributes, triggering conditions, and related information, while eliminating synonyms and supplementing standardized expressions. Then, Transformer encoding technology is used to generate a 32-dimensional feature vector of consultation symptoms and store it in the consultation channel of the feature vector cache.

[0038] The attention-weighted fusion mechanism is used to perform cross-modal fusion calculations on the tongue feature vector from visual examination, the pulse feature vector from palpation, the audio feature vector from auscultation, and the symptom feature vector from inquiry to generate the multimodal fusion feature vector, specifically including: The tongue appearance feature vector, pulse appearance feature vector, auscultation audio feature vector, and symptom inquiry feature vector are mapped to the hidden feature space through a fully connected mapping layer to achieve dimensionality unification, thereby generating the hidden feature vector of the tongue appearance. Pulse diagnosis hidden layer feature vector Auscultation audio hidden layer feature vector and the hidden feature vector of symptoms during medical consultation And summarized into hidden feature vectors of the four diagnostic methods of traditional Chinese medicine. The hidden feature vectors of the four diagnostic methods of traditional Chinese medicine are analyzed using a learnable weight matrix. The feature mapping is performed, and the corresponding calculation formula is as follows: In the formula, Represents the query weight matrix; Key weight matrix; Represents the value weight matrix; Represents the global query vector matrix; Global key vector matrix; Represents a global value vector matrix; This represents the query vectors for tongue appearance by observation, pulse appearance by palpation, audio by auscultation, and symptoms by inquiry. These represent the key vectors for tongue appearance in visual diagnosis, pulse appearance in palpation, audio in auscultation and olfaction, and symptom inquiry, respectively. These represent the value vectors for tongue appearance during inspection, pulse appearance during palpation, audio value vectors during auscultation and olfaction, and symptom value vectors during inquiry, respectively. Based on the global query vector matrix The global key vector matrix The global value vector matrix The hidden feature vectors of the four diagnostic methods in Traditional Chinese Medicine are calculated based on the dot product similarity theory. The corresponding attention weight matrix and the attention weight matrix Perform matrix multiplication operations with different global value vectors respectively, and then summarize to obtain the enhanced feature representation. The enhanced feature representation is obtained by using a weighted fusion formula. Perform a weighted summation to generate the multimodal fusion feature vector. The corresponding calculation formula is as follows: In the formula, This represents the normalized exponential function; Represents the dimension of the global key vector matrix; These respectively represent the enhanced features of tongue appearance in visual diagnosis, enhanced features of pulse in palpation, enhanced audio in auscultation, and enhanced features of symptoms in inquiry. These represent the enhanced feature weights for visual diagnosis of the tongue, palpation of the pulse, auscultation of the audio, and inquiry of symptoms, respectively.

[0039] The learnable weight matrix includes a query weight matrix, a key weight matrix, and a value weight matrix. The query weight matrix enhances the query discriminative power of the hidden feature vectors from the four diagnostic methods of Traditional Chinese Medicine (TCM). The key weight matrix optimizes the relevance measurement dimension of these hidden feature vectors. The value weight matrix preserves the effective information density of the hidden feature vectors. Together, these three components construct a query-key-value attention computation triplet, enabling the attention weighted fusion mechanism to complete the entire process of calculating relevance, assigning weights, and weighted fusion, achieving deep synergy of the hidden feature vectors from the four diagnostic methods of TCM. Attention Weight Matrix This refers to the degree of attention paid by the m-th modality hidden feature vector to the u-th modality hidden feature vector in the four diagnostic methods of Traditional Chinese Medicine. It quantifies the correlation strength between the hidden feature vectors of two modalities, assigning higher attention weights to mutually corroborating hidden feature vectors and lower attention weights to contradictory hidden feature vectors. Enhanced feature vectors refer to sets of enhanced features formed by integrating effective information from hidden feature vectors of other modalities after quantifying the cross-modal correlation strength of the hidden feature vectors of each modality through attention weights. This achieves cross-modal information complementarity and improves the discriminative ability and robustness of the hidden feature vectors.

[0040] Understandably, the tongue image feature vector, pulse image feature vector, auscultation and olfaction audio feature vector, and symptom feature vector of inquiry are read from the tongue image channel, pulse image channel, auscultation and olfaction audio feature vector, and symptom feature vector of inquiry respectively from the feature vector cache channel. The dimensions are then unified through a fully connected mapping layer. The resulting hidden layer feature vectors of tongue image, pulse image, auscultation and olfaction audio, and symptom feature vector of inquiry have the same dimension length, thus eliminating the dimensional barrier of multimodal feature vector collaborative analysis.

[0041] The knowledge base retrieval and encoding module is used to retrieve the medical heritage experience knowledge base and to use vector encoding technology to perform vector encoding on the medical heritage experience knowledge base to generate a medical knowledge embedding matrix. Among them, the Medical Heritage Experience Knowledge Base refers to the collection of heritage experience accumulated by medical schools through long-term clinical practice, including 1,200 classic medical case records and a summary of syndrome differentiation rules. The classic medical case records refer to records of tongue appearance by observation, pulse appearance by palpation, audio records of auscultation and olfaction, records of symptoms by inquiry, and data on syndrome differentiation conclusions. The summary of syndrome differentiation rules refers to the set of logical rules summarized by medical schools based on classic medical case records.

[0042] The process of calling upon a medical heritage experience knowledge base and using vector encoding technology for vector encoding and concatenation to generate a medical knowledge embedding matrix specifically includes: The medical heritage knowledge base includes records of classic medical cases and summaries of diagnostic rules. Named entity recognition technology is used to extract entities from the classic medical records and label the entity categories. Candidate classic medical records are generated by summarizing them. Based on the standard symptom terminology lexicon, the candidate classic medical records are aligned to generate standard classic medical records. The summary of dialectical rules is logically analyzed using rule parsing technology to extract rule logical expressions. Based on the rule logical expressions, the summary of dialectical rules is structurally reconstructed to generate a standard summary of dialectical rules. The vector encoding technology is used to perform vector encoding on the standard classic medical case records and the standard dialectical rule summary, respectively, to generate feature vectors of classic medical case records and feature vectors of dialectical rule summary, which are then horizontally concatenated to generate the medical knowledge embedding matrix.

[0043] For example, a classic medical case record handwritten by an old Chinese medicine doctor in the medical inheritance experience knowledge base is as follows: "Patient Zhang, female, 45 years old, recently suffered from depression, poor mood, chest and rib pain, and fullness in both ribs. Her tongue was pale red with a thin white coating, her pulse was wiry and thready, and she sighed frequently. The diagnosis was liver stagnation and spleen deficiency. She was treated with Xiaoyao San with modifications, Bupleurum 9g, Angelica sinensis 12g, Paeonia lactiflora 15g, Atractylodes macrocephala 10g, Poria cocos 10g, Glycyrrhiza uralensis 3g, Mentha haplocalyx 3g, and 2 slices of ginger."

[0044] Named entity recognition technology was used to extract and label candidate classic medical records, including patient information, tongue diagnosis records, and recommended prescriptions. Based on a standard symptom terminology lexicon, entity alignment was performed on these candidate classic medical records to generate standard classic medical records, as shown in Table 1.

[0045] The rule parsing technology is used to perform logical association analysis on the summary of the diagnostic rules, and the corresponding rule logical expression is extracted as follows: IF Tongue appearance by observation = "pale red with thin white coating" AND Pulse appearance by palpation = "wiry and thin" AND Auscultation and olfaction = "frequent sighing" AND Symptoms by inquiry = "emotional distress, chest and rib distension"; THEN "Diagnostic conclusion data = liver stagnation and spleen deficiency".

[0046] The summary of dialectical rules is restructured based on rule-based logical expressions to generate a standard summary of dialectical rules, as shown in Table 2:

[0047] By using the above methods, the experience of medical schools of thought is transformed into a medical knowledge embedding matrix that is adapted to and optimized for learning and analysis of dialectical reasoning models, thus solving the problems of difficulty in inheriting and quantifying traditional medical experience.

[0048] The model building and dialectical reasoning module is used to perform deep feature extraction and multi-task classification prediction on the multimodal fusion feature vector based on the medical knowledge embedding matrix and an optimized dialectical reasoning model to generate dialectical conclusion data. Based on the medical knowledge embedding matrix, an optimized dialectical reasoning model is used to perform deep feature extraction and multi-task classification prediction on the multimodal fusion feature vector to generate dialectical conclusion data, specifically including: The optimized dialectical reasoning model, which cascades an encoder and decoder, integrates multimodal feature vectors. Medical knowledge embedding matrix Input to encoder and calculate cosine similarity The corresponding calculation formula is as follows: In the formula, Represents L2 norm operations; A preset cosine similarity threshold is set, when the cosine similarity... If the cosine similarity is greater than or equal to the cosine similarity threshold, then the cosine similarity is filtered. The classic medical case record feature vector corresponding to the medical knowledge embedding matrix is ​​used, and the selected classic medical case record feature vector is horizontally concatenated with the multimodal fusion feature vector to generate a knowledge-enhanced feature vector; The fully connected classification head network in the decoder is used to perform parallel inference on the knowledge-enhanced feature vector to output the probability distribution of constitution type, syndrome type, and risk level. Combined with the maximum value sampling strategy, the corresponding constitution type prediction result, syndrome type prediction result, and risk level prediction result are output. The constitution type prediction result, syndrome type prediction result, and risk level prediction result are integrated and encapsulated to generate the dialectical conclusion data.

[0049] Among them, the knowledge-enhanced feature vector refers to the fused feature vector generated by horizontally concatenating the multimodal fusion feature vector with the classic medical case record feature vectors obtained from screening and having a similarity greater than or equal to the cosine similarity threshold. This achieves deep fusion of the multimodal fusion feature vector and the classic medical case record feature vector, improving the accuracy and interpretability of the optimized dialectical reasoning model. The constitution type prediction result refers to the patient's constitution classification, including balanced constitution, qi deficiency constitution, yang deficiency constitution, yin deficiency constitution, phlegm-dampness constitution, damp-heat constitution, blood stasis constitution, qi stagnation constitution, and special constitution. The syndrome type prediction result refers to the patient's pathological state classification, including liver qi stagnation and spleen deficiency syndrome, spleen and stomach deficiency-cold syndrome, and phlegm-dampness obstructing the lungs. The risk level prediction result refers to the patient's health risk classification, including low risk, medium risk, and high risk.

[0050] The optimized dialectical reasoning model is deployed on a cloud server and configured with an Intel Xeon Gold 6338 processor, 128GB of memory, and an NVIDIA A100 graphics card. The encoder of the optimized dialectical reasoning model calculates the cosine similarity between the multimodal fusion feature vector and the medical knowledge embedding matrix. Classic medical case record feature vectors with a cosine similarity greater than or equal to the cosine similarity threshold are selected and horizontally concatenated with the multimodal fusion feature vector to generate a knowledge-enhanced feature vector. This allows the classic medical case record feature vector that is closest to the multimodal fusion feature vector of the current patient to be retrieved. This ensures that the reused classic medical case records are highly matched with the patient's actual constitution and pathological state, rather than generalized general knowledge, making the reasoning process of the optimized dialectical reasoning model transparent and traceable.

[0051] The decoder of the optimized dialectical reasoning model consists of three independent fully connected classifiers, corresponding to the constitution type prediction task, syndrome type prediction task, and risk level prediction task, respectively. Each fully connected classifier contains two layers of fully connected classifier networks, corresponding to the output probability distributions of constitution type, syndrome type, and risk level, respectively. The maximum value sampling strategy is used to select the category with the highest probability value from the probability distributions of constitution type, syndrome type, and risk level, respectively, and these are used as the prediction results of constitution type, syndrome type, and risk level, respectively, and integrated to generate dialectical conclusion data. This ensures that the reasoning time of the optimized dialectical reasoning model to deduce the dialectical conclusion data is controlled within 0.8 seconds, and the reasoning accuracy of the dialectical conclusion data is improved by more than 40% compared with the single diagnostic reasoning model, with a consistency rate of 92% with the dialectical conclusion data of the physician's diagnosis.

[0052] The scheme matching and interactive correction module is used to query the prescription knowledge base and the nursing rule base, match the recommended prescription scheme and specific nursing scheme corresponding to the syndrome differentiation conclusion data based on the index mapping relationship table, summarize and encapsulate the syndrome differentiation conclusion data, the recommended prescription scheme and the specific nursing scheme, generate a TCM four diagnostic auxiliary treatment scheme and push it to the interactive platform for physicians to refer to and correct.

[0053] The query formula knowledge base and nursing rule base, based on the index mapping relationship table, match the recommended formula schemes and specific nursing schemes corresponding to the syndrome differentiation conclusion data, specifically including: Based on the diagnostic conclusion data, the formula knowledge base is queried, and the syndrome type prediction result is used as the index to search in the syndrome-formula mapping table to obtain multiple candidate formula schemes; Based on the syndrome type prediction results and the candidate prescription schemes, the cosine similarity algorithm is used to calculate the prescription matching score. Based on the predicted constitution type and the candidate prescriptions, the cosine similarity algorithm is used to calculate the constitution matching score. ; The formula will be matched and scored. Matching score with the physical condition The weighted fusion is used to obtain the comprehensive score of the prescription. In the formula, This indicates the weight of the prescription matching score; The weight of the constitution matching score is indicated by the highest comprehensive score of the prescribed formula. The corresponding candidate prescription scheme is used as the first prescription scheme; The formula addition and subtraction rule library is retrieved, and the tongue appearance feature vector, pulse appearance feature vector, auscultation and olfaction audio feature vector, and symptom feature vector are input into the formula addition and subtraction condition expression for logical judgment. If the output logical judgment result is true, the feature vector-formula addition and subtraction mapping table is triggered to perform the corresponding formula addition and subtraction operation and update the first formula scheme to generate the second formula scheme. The second prescription scheme is input into the drug compatibility data table and a group of prescriptions in the second prescription scheme is randomly traversed. If the group of prescriptions exists in the drug compatibility data table, a compatibility warning message and a prescription replacement scheme are generated. The compatibility warning information, the prescription replacement scheme, and the set of prescriptions are returned to the prescription addition and subtraction rule library, and prescription addition and subtraction operations are performed on the set of prescriptions according to the prescription replacement scheme to obtain the recommended prescription scheme; The syndrome type prediction results and the constitution type prediction results are input into the nursing care rule base, and the specific nursing care plan is obtained based on the syndrome-nursing care mapping table and the constitution-nursing care mapping table.

[0054] The first prescription scheme refers to the basic prescription scheme selected based on the predicted results of syndrome type and constitution type. It includes a list of prescription names, a list of constituent drug names, standard dosage ranges for each drug, a description of the compatibility structure, and a list of the main syndrome types it treats. The second prescription scheme refers to the personalized prescription scheme updated by adding, removing, or adjusting the dosage of drugs from the first prescription scheme. The recommended prescription scheme refers to the clinically usable scheme generated after three levels of optimization: basic screening, personalized additions and subtractions, and safety verification. Specific care plans include dietary therapy plans, acupoint health care plans, and lifestyle regulation suggestions. The dietary therapy plan includes a list of recommended foods, a list of prohibited foods, and a sample recipe description; the acupoint health care plan includes a list of recommended acupoints and methods of acupoint manipulation; and the lifestyle regulation suggestions include suggestions for adjusting daily routines, exercise methods, and emotional regulation.

[0055] Specifically, the system extracts the syndrome type prediction results from the diagnostic conclusion data as an index key, queries the syndrome-prescription mapping table in the prescription knowledge base, and extracts all candidate prescription schemes associated with the syndrome-prescription mapping table, thus achieving intelligent mapping from diagnostic conclusion data to candidate prescription schemes. The cosine similarity algorithm is used to calculate the prescription matching score between the syndrome type prediction results and the candidate prescription schemes, and the cosine similarity algorithm is used to calculate the constitution matching score between the constitution type prediction results and the candidate prescription schemes. A higher prescription matching score indicates a stronger targeting of the candidate prescription scheme to the syndrome type prediction results, and a higher constitution matching score indicates a better fit between the candidate prescription scheme and the patient's constitution type prediction results. This approach balances the targeting of the syndrome type prediction results with the suitability of the constitution type prediction results, avoiding mismatches in candidate prescription schemes caused by single-dimensional screening.

[0056] Furthermore, the formula matching score and constitution matching score are weighted and fused according to preset weights to calculate the comprehensive formula score corresponding to each candidate formula scheme. All comprehensive formula scores are sorted in descending order, and the candidate formula scheme with the highest comprehensive formula score is selected as the first formula scheme, thereby balancing the treatment needs of syndrome type prediction results and the adaptation needs of constitution type prediction results.

[0057] Understandably, performing prescription adjustments on the first treatment plan requires considering the patient's current constitution type and the corresponding concurrent and altered symptoms predicted. Therefore, it is necessary to accurately capture concurrent and altered symptoms based on the patient's tongue appearance feature vector, pulse appearance feature vector, auscultation and olfaction audio feature vector, and symptom inquiry feature vector. The prescription adjustment rule library is retrieved; each rule includes a conditional expression and a corresponding adjustment operation. Key parameters are extracted from the tongue appearance feature vector, pulse appearance feature vector, auscultation and olfaction audio feature vector, and symptom inquiry feature vector. These key parameters are substituted into the conditional expression for logical judgment. If the logical judgment result is true, the corresponding prescription adjustment rule is triggered and the adjustment operation is executed. This transforms the tongue appearance feature vector, pulse appearance feature vector, auscultation and olfaction audio feature vector, and symptom inquiry feature vector into specific prescription adjustment operation instructions, resulting in a personalized second prescription plan, achieving the goal of precise diagnosis and treatment tailored to each individual.

[0058] Finally, the drug incompatibility data table is called to perform safety verification and compliance checks on the second prescription scheme. A set of prescriptions in the second prescription scheme is randomly traversed and checked to see if they exist in the drug incompatibility data table. If they exist in the drug incompatibility data table, alternative drugs with similar efficacy and no drug incompatibility are retrieved. Incompatibility warning information and prescription replacement schemes are generated, and drug replacement is performed through prescription addition and subtraction operations, thereby avoiding the medication safety risks caused by drug incompatibility and ensuring the clinical use safety of the generated recommended prescription scheme.

[0059] After the TCM four diagnostic methods auxiliary treatment plan is pushed to the interactive platform for physicians to refer to and revise, it also includes: Physicians can view the TCM four diagnostic methods auxiliary treatment plan through the visual interface of the interactive platform and make manual corrections to generate the final TCM four diagnostic methods auxiliary treatment plan; The correction content, correction tags, and the final TCM four diagnostic methods auxiliary treatment plan recorded by the interactive platform are transmitted to the optimized syndrome differentiation reasoning model, and an incremental learning algorithm is used for self-updating and optimization.

[0060] Furthermore, given the complexity and variability of clinical scenarios and the dynamic iteration of medical experience, the system aggregates and encapsulates diagnostic conclusions, recommended prescriptions, and specific care plans to generate TCM four diagnostic methods-assisted treatment plans, which are then pushed to the interactive platform. By constructing a human-machine collaborative interactive correction and incremental learning mechanism, a closed-loop fusion of medical experience and optimized diagnostic reasoning models is achieved, enabling the system to have continuous learning and self-evolution capabilities, significantly improving the level of TCM diagnosis and treatment and service quality. Example 2

[0061] like Figure 2 The image shows a TCM diagnostic aid method based on multimodal feature fusion, provided by an embodiment of the present invention. The method includes: S1. Obtain the patient's original data of the four diagnostic methods of traditional Chinese medicine and preprocess it to generate data of the four diagnostic methods of traditional Chinese medicine. The data of the four diagnostic methods of traditional Chinese medicine includes tongue image data of inspection, pulse waveform data of palpation, audio signal data of auscultation, and text data of symptoms of inquiry. S2, image processing technology, wavelet packet decomposition algorithm, Mel-frequency cepstral coefficient feature extraction algorithm, and semantic coding technology are used to extract features and transform vectors from the visual tongue image data, the palpation pulse waveform data, the auscultation audio signal data, and the inquiry symptom text data, respectively. Cross-modal fusion calculation is performed through attention weighted fusion mechanism to generate multimodal fusion feature vectors. S3, invoke the medical inheritance experience knowledge base, and use vector encoding technology to perform vector encoding on the medical inheritance experience knowledge base to generate a medical knowledge embedding matrix; S4. Based on the medical knowledge embedding matrix, an optimized dialectical reasoning model is used to perform deep feature extraction and multi-task classification prediction on the multimodal fusion feature vector to generate dialectical conclusion data. S5. Query the formula knowledge base and nursing rule base, match the recommended formula scheme and specific nursing scheme corresponding to the syndrome differentiation conclusion data based on the index mapping relationship table, summarize and encapsulate the syndrome differentiation conclusion data, the recommended formula scheme and the specific nursing scheme, generate a TCM four diagnostic auxiliary treatment plan and push it to the interactive platform for physicians to refer to and correct.

[0062] Through the above embodiments, this invention uses a data acquisition and preprocessing module to acquire and preprocess the patient's original TCM four diagnostic methods data, generating TCM four diagnostic data, which includes tongue image data from inspection, pulse waveform data from palpation, audio signal data from auscultation, and text data of symptoms from inquiry. A feature extraction and fusion module uses image processing technology, wavelet packet decomposition algorithm, Mel-frequency cepstral coefficient feature extraction algorithm, and semantic coding technology to extract features and transform vectors from the tongue image data from inspection, pulse waveform data from palpation, audio signal data from auscultation, and text data of symptoms from inquiry, respectively. An attention-weighted fusion mechanism is used for cross-modal fusion calculation to generate a multimodal fusion feature vector. The invention also utilizes a knowledge base call and... The encoding module is used to call upon the medical heritage experience knowledge base and use vector encoding technology to vectorize the medical heritage experience knowledge base to generate a medical knowledge embedding matrix. The model building and dialectical reasoning module is used to perform deep feature extraction and multi-task classification prediction on multimodal fusion feature vectors based on the medical knowledge embedding matrix and using an optimized dialectical reasoning model to generate dialectical conclusion data. The scheme matching and interactive correction module is used to query the prescription knowledge base and the nursing rule base, match the recommended prescription scheme and specific nursing scheme corresponding to the dialectical conclusion data based on the index mapping relationship table, summarize and encapsulate the dialectical conclusion data, recommended prescription scheme and specific nursing scheme, generate a TCM four diagnostic auxiliary treatment scheme and push it to the interactive platform for physicians to refer to and correct.

[0063] This invention achieves unified representation and deep fusion of tongue image data from visual examination, pulse waveform data from palpation, audio signal data from auscultation, and textual data from symptom inquiry by constructing a standardized processing and feature extraction mechanism for multimodal diagnostic data. This overcomes the limitations of existing technologies that process data independently from a single diagnostic method, effectively improving the accuracy and completeness of collaborative analysis of multimodal diagnostic data. Furthermore, by deeply embedding the inherited experience of medical schools into an optimized diagnostic reasoning model using vector coding technology, it enables dynamic retrieval and knowledge enhancement of valuable clinical knowledge such as classic medical cases and diagnostic rules, significantly improving the clinical matching degree and theoretical solvability of diagnostic conclusion data. The system is characterized by its comprehensive nature. By establishing a full-chain solution generation mechanism encompassing "syndrome differentiation, prescription, and nursing care," it achieves a complete output of TCM four-diagnosis auxiliary treatment plans, from constitution prediction, syndrome judgment, risk prediction, prescription matching, personalized addition and subtraction to nursing care suggestions. Furthermore, it incorporates a drug compatibility data table to detect and ensure medication safety, significantly enhancing the systematicness, personalization, and practicality of TCM four-diagnosis auxiliary treatment plans. Through the construction of a human-machine collaborative interactive correction and incremental learning mechanism, it achieves a closed-loop iteration of physician correction feedback and optimization of the syndrome differentiation reasoning model, enabling the system to continuously learn and self-evolve, effectively adapting to the dynamic evolution of clinical needs.

[0064] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0065] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, including read-only memory (ROM), random access memory (RAM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), one-time programmable read-only memory (OTPROM), electrically-Erasable Programmable Read-Only Memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, disk storage, magnetic tape storage, or any other computer-readable medium capable of carrying or storing data.

[0066] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

Claims

1. A TCM diagnostic system based on multimodal feature fusion, characterized in that, The system includes: The data acquisition and preprocessing module is used to acquire the patient's original data of the four diagnostic methods of traditional Chinese medicine and preprocess it to generate data of the four diagnostic methods of traditional Chinese medicine. The data of the four diagnostic methods of traditional Chinese medicine includes tongue image data of inspection, pulse waveform data of palpation, audio signal data of auscultation, and text data of symptoms of inquiry. The feature extraction and fusion module is used to extract features and transform vectors from the visual tongue image data, palpation pulse waveform data, auscultation audio signal data, and inquiry symptom text data using image processing technology, wavelet packet decomposition algorithm, Mel-frequency cepstral coefficient feature extraction algorithm, and semantic coding technology, respectively. It also performs cross-modal fusion calculation through an attention-weighted fusion mechanism to generate a multimodal fusion feature vector. The knowledge base retrieval and encoding module is used to retrieve the medical heritage experience knowledge base and to use vector encoding technology to perform vector encoding on the medical heritage experience knowledge base to generate a medical knowledge embedding matrix. The model building and dialectical reasoning module is used to perform deep feature extraction and multi-task classification prediction on the multimodal fusion feature vector based on the medical knowledge embedding matrix and an optimized dialectical reasoning model to generate dialectical conclusion data. The scheme matching and interactive correction module is used to query the prescription knowledge base and the nursing rule base, match the recommended prescription scheme and specific nursing scheme corresponding to the syndrome differentiation conclusion data based on the index mapping relationship table, summarize and encapsulate the syndrome differentiation conclusion data, the recommended prescription scheme and the specific nursing scheme, generate a TCM four diagnostic auxiliary treatment scheme and push it to the interactive platform for physicians to refer to and correct.

2. The TCM four-diagnostic auxiliary system based on multimodal feature fusion according to claim 1, characterized in that, The process involves acquiring and preprocessing the patient's original data from the four diagnostic methods of Traditional Chinese Medicine (TCM) to generate TCM diagnostic data. This TCM diagnostic data includes tongue image data from inspection, pulse waveform data from palpation, audio signal data from auscultation, and textual data of symptoms from inquiry. Specifically, it includes: The ring-shaped standardized light source component is dynamically adjusted according to the patient's posture, and the patient's oral cavity image data is acquired through a high-definition camera component. The edge detection algorithm is used to detect the edge contour of the patient's tongue, and the internal area of ​​the edge contour of the patient's tongue is preserved. Then, the light correction algorithm is used for illumination correction and the median filtering algorithm is used for noise removal to generate the tongue image data for visual diagnosis. A pressure sensor array is linearly arranged against the radial artery of the patient's wrist to collect pulse pulsation signals and transmit them to a signal conditioning circuit. The pulse pulsation signals are then amplified, noise filtered, and converted to digital values ​​by a signal amplifier, a low-pass filter, and an analog-to-digital converter in the signal conditioning circuit to generate the palpation pulse waveform data. The patient is guided to read and breathe deeply according to the preset standard reading content through the preset display unit. The reading voice signal and breathing audio signal are collected by the cardioid pickup unit in the directional microphone assembly and transmitted to the audio processing circuit. The reading voice signal and breathing audio signal are amplified, high-frequency component filtered out and analog-to-digital converted in sequence through the preamplifier, anti-aliasing filter and analog-to-digital converter in the audio processing circuit, and the data is summarized to generate the auscultation audio signal data. The patient's symptom characters are collected using a structured symptom questionnaire component, and the data is cleaned using regular expressions to retain valid patient symptom characters and generate the consultation symptom text data.

3. The TCM four-diagnostic auxiliary system based on multimodal feature fusion according to claim 1, characterized in that, The process employs image processing techniques, wavelet packet decomposition algorithms, Mel-frequency cepstral coefficient feature extraction algorithms, and semantic coding techniques to perform feature extraction and vector transformation on the visual examination tongue image data, the palpation pulse waveform data, the auscultation audio signal data, and the inquiry symptom text data, specifically including: The RGB color space of the tongue image data is split using channel separation technology to obtain the R-channel pixel grayscale matrix, G-channel pixel grayscale matrix, and B-channel pixel grayscale matrix. Mathematical statistical calculations are then performed to obtain the mean, variance, and skewness of the R-channel, G-channel, and B-channel, and the color distribution characteristics are summarized to generate the color distribution features. The gray-level co-occurrence matrix algorithm is used to calculate the gray-level co-occurrence relationship corresponding to the R-channel pixel gray-level matrix, the G-channel pixel gray-level matrix, and the B-channel pixel gray-level matrix, respectively. The texture roughness, texture uniformity, and texture directionality in the corresponding R-channel gray-level co-occurrence matrix, G-channel gray-level co-occurrence matrix, and B-channel gray-level co-occurrence matrix are generated and extracted, and the texture structure features are summarized to generate texture structure features. Using geometric morphology quantization technology, the area, perimeter, aspect ratio, and edge curvature changes of the tongue image data are calculated and summarized to generate geometric morphological features; The color distribution features, texture structure features and geometric morphology features are sequentially concatenated, and dimensionality reduction optimization is performed using principal component analysis algorithm to generate a tongue image feature vector for visual diagnosis. For the pulse waveform data, a high-pass filtering algorithm is used to perform baseline drift correction, and an adaptive thresholding algorithm is used to detect the position of the main peak and perform waveform period segmentation to obtain multiple single-period pulse waveform data. Based on the time-domain feature extraction algorithm, features are extracted from the multiple single-period pulse waveform data to obtain the amplitude value of the main peak. The slope of the rising segment of the main wave The slope of the main wave's descending segment Ratio of tidal peak amplitude to main wave peak amplitude The ratio of the amplitude of the diphthral peak to the amplitude of the main peak and the duration of single-cycle pulse waveform data The corresponding calculation formula is as follows: In the formula, Indicates the peak amplitude of the tidal wave; This indicates the amplitude of the diphtheria peak; This represents the amplitude value of the single-cycle pulse waveform data at time t; These represent the start and end times of a single-cycle pulse waveform data, respectively. This indicates the time t corresponding to the amplitude value of the main wave peak; This represents the amplitude value of the single-cycle pulse waveform data at time t corresponding to the i-th sampling point; This represents the amplitude value of the single-cycle palpation pulse waveform data at time t corresponding to the (i-1)th sampling point; Indicates the time interval between adjacent sampling points; Calculate and integrate the average amplitude of the main peak of the palpation pulse waveform data from multiple single cycles. Mean slope of the rising segment of the main wave Mean slope of the descending segment of the main wave Mean ratio of tidal peak amplitude to main wave peak amplitude Mean ratio of the amplitude of the diphthral peak to the amplitude of the main peak The average duration of single-cycle pulse waveform data Generate temporal features of pulse diagnosis ; The wavelet packet decomposition algorithm is used to perform multi-scale time-frequency decomposition on multiple single-cycle palpation pulse waveform data to obtain palpation pulse component signals. And extract the palpation pulse component signal. Corresponding energy percentage and clock frequency parameters Calculate and integrate the average energy percentage Average value of main frequency parameters Generate frequency domain features of palpation pulse. The corresponding calculation formula is as follows: In the formula, j represents the number of decomposition layers; k represents the frequency band number; This represents the decomposition of low-pass filter coefficients; This represents the decomposition of the high-pass filter coefficients; n represents the coefficient index; Indicates Fast Fourier Transform; This indicates the value of the independent variable corresponding to the maximum value; The time-domain features of the palpation pulse The frequency domain features of the palpation pulse The components are sequentially concatenated and then optimized for dimensionality reduction using the principal component analysis algorithm to generate a pulse diagnosis feature vector. The auscultation audio signal data is segmented using a sliding window framing technique to obtain multiple auscultation audio signal frames. The multiple auscultation audio signal frames are then weighted using a Hamming window function to obtain an auscultation audio signal enhancement frame. The enhanced frames of the auscultation audio signal are converted into an auscultation audio spectrum sequence by the Fast Fourier Transform and classified according to the frequency interval differences to obtain the reading speech spectrum sequence and the breathing audio spectrum sequence. The reading speech spectrum sequence and the breathing audio spectrum sequence are input into the Mel filter bank, and weighted filtering, logarithmic operation and discrete cosine transform are performed based on the Mel cepstral coefficient feature extraction algorithm to obtain the static features and dynamic features of the reading speech, the energy distribution features of the breathing spectrum and the rhythmic features of the breathing cycle. The static features of the reading speech, the dynamic features of the reading speech, the energy distribution features of the breathing spectrum and the rhythmic features of the breathing cycle are concatenated in sequence, and the dimensionality reduction optimization is performed by the principal component analysis algorithm to generate the auscultation audio feature vector. The text data of the consultation symptoms is mapped to the semantic data of the consultation symptoms according to the text-semantic mapping rules. The semantic data of the consultation symptoms is segmented and aligned with entities based on the standard lexicon of symptom terms. Then, the feature vector of the consultation symptoms is generated by using Transformer encoding technology.

4. The TCM four-diagnostic auxiliary system based on multimodal feature fusion according to claim 3, characterized in that, The attention-weighted fusion mechanism is used to perform cross-modal fusion calculations on the tongue feature vector from visual examination, the pulse feature vector from palpation, the audio feature vector from auscultation, and the symptom feature vector from inquiry to generate the multimodal fusion feature vector, specifically including: The tongue appearance feature vector, pulse appearance feature vector, auscultation audio feature vector, and symptom inquiry feature vector are mapped to the hidden feature space through a fully connected mapping layer to achieve dimensionality unification, thereby generating the hidden feature vector of the tongue appearance. Pulse diagnosis hidden layer feature vector Auscultation audio hidden layer feature vector and the hidden feature vector of symptoms during medical consultation And summarized into hidden feature vectors of the four diagnostic methods of traditional Chinese medicine. The hidden feature vectors of the four diagnostic methods of traditional Chinese medicine are analyzed using a learnable weight matrix. The feature mapping is performed, and the corresponding calculation formula is as follows: In the formula, Represents the query weight matrix; Represents the key weight matrix; Represents the value weight matrix; Represents the global query vector matrix; Represents the global key vector matrix; Represents a global value vector matrix; These represent the query vectors for tongue appearance by observation, pulse appearance by palpation, audio by auscultation, and symptoms by inquiry, respectively. These represent the key vectors for tongue appearance in visual diagnosis, pulse appearance in palpation, audio in auscultation and olfaction, and symptom inquiry, respectively. These represent the value vectors for tongue appearance during inspection, pulse appearance during palpation, audio value vectors during auscultation and olfaction, and symptom value vectors during inquiry, respectively. Based on the global query vector matrix The global key vector matrix The global value vector matrix The hidden feature vectors of the four diagnostic methods in Traditional Chinese Medicine are calculated based on the dot product similarity theory. The corresponding attention weight matrix and the attention weight matrix Perform matrix multiplication operations with different global value vectors respectively, and then summarize to obtain the enhanced feature representation. The enhanced feature representation is obtained by using a weighted fusion formula. Perform a weighted summation to generate the multimodal fusion feature vector. The corresponding calculation formula is as follows: In the formula, This represents the normalized exponential function; Represents the dimension of the global key vector matrix; These respectively represent the enhanced features of tongue appearance in visual diagnosis, enhanced features of pulse in palpation, enhanced audio in auscultation, and enhanced features of symptoms in inquiry. These represent the enhanced feature weights for visual diagnosis of the tongue, palpation of the pulse, auscultation of the audio, and inquiry of symptoms, respectively.

5. The TCM four-diagnostic auxiliary system based on multimodal feature fusion according to claim 1, characterized in that, The process involves calling upon a medical heritage experience knowledge base and using vector encoding technology for vector encoding and concatenation to generate a medical knowledge embedding matrix. Specifically... include: The medical heritage knowledge base includes records of classic medical cases and summaries of diagnostic rules. Named entity recognition technology is used to extract entities from the classic medical records and label the entity categories. Candidate classic medical records are generated by summarizing them. Based on the standard symptom terminology lexicon, the candidate classic medical records are aligned to generate standard classic medical records. The summary of dialectical rules is logically analyzed using rule parsing technology to extract rule logical expressions. Based on the rule logical expressions, the summary of dialectical rules is structurally reconstructed to generate a standard summary of dialectical rules. The vector encoding technology is used to perform vector encoding on the standard classic medical case records and the standard dialectical rule summary, respectively, to generate feature vectors of classic medical case records and feature vectors of dialectical rule summary, which are then horizontally concatenated to generate the medical knowledge embedding matrix.

6. The TCM four-diagnostic auxiliary system based on multimodal feature fusion according to claim 1, characterized in that, Based on the medical knowledge embedding matrix, an optimized dialectical reasoning model is used to perform deep feature extraction and multi-task classification prediction on the multimodal fusion feature vector to generate dialectical conclusion data, specifically including: The optimized dialectical reasoning model, which cascades an encoder and decoder, integrates multimodal feature vectors. Medical knowledge embedding matrix Input to encoder and calculate cosine similarity The corresponding calculation formula is as follows: In the formula, Represents L2 norm operations; A preset cosine similarity threshold is set, when the cosine similarity... If the cosine similarity is greater than or equal to the cosine similarity threshold, then the cosine similarity is filtered. The classic medical case record feature vector corresponding to the medical knowledge embedding matrix is ​​used, and the selected classic medical case record feature vector is horizontally concatenated with the multimodal fusion feature vector to generate a knowledge-enhanced feature vector; The fully connected classification head network in the decoder is used to perform parallel inference on the knowledge-enhanced feature vector to output the probability distribution of constitution type, syndrome type, and risk level. Combined with the maximum value sampling strategy, the corresponding constitution type prediction result, syndrome type prediction result, and risk level prediction result are output. The constitution type prediction result, syndrome type prediction result, and risk level prediction result are integrated and encapsulated to generate the dialectical conclusion data.

7. A TCM diagnostic system based on multimodal feature fusion for auxiliary diagnosis and treatment according to claim 3 or 6, characterized in that, The query formula knowledge base and nursing rule base, based on the index mapping relationship table, match the recommended formula schemes and specific nursing schemes corresponding to the syndrome differentiation conclusion data, specifically including: Based on the diagnostic conclusion data, the formula knowledge base is queried, and the syndrome type prediction result is used as the index to search in the syndrome-formula mapping table to obtain multiple candidate formula schemes; Based on the syndrome type prediction results and the candidate prescription schemes, the cosine similarity algorithm is used to calculate the prescription matching score. Based on the predicted constitution type and the candidate prescriptions, the cosine similarity algorithm is used to calculate the constitution matching score. ; The formula will be matched and scored. Matching score with the physical condition The weighted fusion is used to obtain the comprehensive score of the prescription. In the formula, This indicates the weight of the prescription matching score; The weight of the constitution matching score is indicated by the highest comprehensive score of the prescribed formula. The corresponding candidate prescription scheme is used as the first prescription scheme; The formula addition and subtraction rule library is retrieved, and the tongue appearance feature vector, pulse appearance feature vector, auscultation and olfaction audio feature vector, and symptom feature vector are input into the formula addition and subtraction condition expression for logical judgment. If the output logical judgment result is true, the feature vector-formula addition and subtraction mapping table is triggered to perform the corresponding formula addition and subtraction operation and update the first formula scheme to generate the second formula scheme. The second prescription scheme is input into the drug compatibility data table and a group of prescriptions in the second prescription scheme is randomly traversed. If the group of prescriptions exists in the drug compatibility data table, a compatibility warning message and a prescription replacement scheme are generated. The compatibility warning information, the prescription replacement scheme, and the set of prescriptions are returned to the prescription addition and subtraction rule library, and prescription addition and subtraction operations are performed on the set of prescriptions according to the prescription replacement scheme to obtain the recommended prescription scheme; The syndrome type prediction results and the constitution type prediction results are input into the nursing care rule base, and the specific nursing care plan is obtained based on the syndrome-nursing care mapping table and the constitution-nursing care mapping table.

8. The TCM four-diagnostic auxiliary system based on multimodal feature fusion according to claim 1, characterized in that, After the TCM four diagnostic methods auxiliary treatment plan is pushed to the interactive platform for physicians to refer to and revise, it also includes: Physicians can view the TCM four diagnostic methods auxiliary treatment plan through the visual interface of the interactive platform and make manual corrections to generate the final TCM four diagnostic methods auxiliary treatment plan; The correction content, correction tags, and the final TCM four diagnostic methods auxiliary treatment plan recorded by the interactive platform are transmitted to the optimized syndrome differentiation reasoning model, and an incremental learning algorithm is used for self-updating and optimization.

9. A TCM four-diagnostic auxiliary treatment method based on multimodal feature fusion, applied to the TCM four-diagnostic auxiliary treatment method based on multimodal feature fusion as described in any one of claims 1-8, characterized in that, The method includes: The patient's original data from the four diagnostic methods of traditional Chinese medicine (TCM) are acquired and preprocessed to generate TCM diagnostic data, which includes tongue image data from inspection, pulse waveform data from palpation, audio signal data from auscultation, and text data of symptoms from inquiry. Image processing technology, wavelet packet decomposition algorithm, Mel-frequency cepstral coefficient feature extraction algorithm, and semantic coding technology are used to extract features and transform vectors from the visual tongue image data, palpation pulse waveform data, auscultation audio signal data, and inquiry symptom text data, respectively. Cross-modal fusion calculation is performed through attention-weighted fusion mechanism to generate multimodal fusion feature vectors. The medical heritage experience knowledge base is invoked, and vector encoding technology is used to vector encode the medical heritage experience knowledge base to generate a medical knowledge embedding matrix; Based on the medical knowledge embedding matrix, an optimized dialectical reasoning model is used to perform deep feature extraction and multi-task classification prediction on the multimodal fusion feature vector to generate dialectical conclusion data. The system queries the formula knowledge base and nursing rule base, matches the recommended formula schemes and specific nursing schemes corresponding to the syndrome differentiation data based on the index mapping relationship table, summarizes and encapsulates the syndrome differentiation data, the recommended formula schemes and the specific nursing schemes, generates a TCM four diagnostic and treatment auxiliary treatment plan, and pushes it to the interactive platform for physicians to refer to and correct.

Images