Traditional Chinese medicine prescription retrieval method and system based on multi-modal traditional Chinese medicine prescription data

Abstract

The application provides a traditional Chinese medicine prescription retrieval method and system based on multi-modal traditional Chinese medicine prescription data, and relates to the technical field of intelligent medical treatment. First, a multi-modal traditional Chinese medicine prescription data set containing traditional Chinese medical record text data, tongue image data, pulse diagnosis image data and voice description data is obtained, and then data preprocessing is performed. Then, feature extraction is performed on the preprocessed data to generate a text semantic, tongue image and pulse diagnosis image feature set, and a multi-modal traditional Chinese medicine prescription feature database is constructed based on the feature set. When receiving input multi-modal query data, the database is called to perform multi-modal feature matching operation, and finally a traditional Chinese medicine prescription retrieval result set is generated and output, so that multi-modal data can be integrated to improve the accuracy and comprehensiveness of traditional Chinese medicine prescription retrieval.

Description

Technical Field

[0001] This invention relates to the field of smart healthcare technology, and more specifically, to a method and system for retrieving traditional Chinese medicine prescriptions based on multimodal traditional Chinese medicine prescription data. Background Technology

[0002] In the field of Traditional Chinese Medicine (TCM), TCM prescription retrieval is of great significance for doctors' clinical reference, teaching and research, and patient information retrieval. Traditional TCM prescription retrieval methods mainly rely on single textual information, such as searching solely based on textual descriptions in TCM medical records. This approach has many limitations, because TCM diagnosis is a comprehensive judgment process, and a single textual description cannot fully encompass the characteristics of a patient's condition.

[0003] On the one hand, single-text data may suffer from inaccurate or incomplete descriptions. Doctors may omit crucial information when recording medical records, or differences in expression habits among different doctors may lead to poor consistency in textual information, thus affecting the accuracy of retrieval. On the other hand, traditional Chinese medicine diagnosis emphasizes the comprehensive use of multiple methods such as observation, auscultation, inquiry, and palpation. Information such as tongue appearance and pulse diagnosis are crucial for judging the condition and prescribing medication, but traditional text retrieval methods cannot fully utilize this non-textual information, making the retrieval results insufficient to meet actual needs. Summary of the Invention

[0004] In view of the aforementioned problems, and in conjunction with the first aspect of the present invention, embodiments of the present invention provide a method for retrieving traditional Chinese medicine prescriptions based on multimodal traditional Chinese medicine prescription data, the method comprising:

[0005] A multimodal TCM prescription data set is obtained, which includes TCM medical record text data, tongue image data, pulse diagnosis image data, and voice description data;

[0006] The multimodal TCM prescription dataset is preprocessed to obtain a preprocessed multimodal TCM prescription dataset.

[0007] Feature extraction is performed on the preprocessed multimodal TCM prescription data set to generate a text semantic feature set, a tongue image feature set, and a pulse diagnosis image feature set;

[0008] A multimodal TCM prescription feature database is constructed based on the text semantic feature set, the tongue image feature set, and the pulse diagnosis image feature set.

[0009] In response to the input multimodal query data, the multimodal TCM prescription feature database is invoked to perform multimodal feature matching operations, generate a set of TCM prescription retrieval results, and output them.

[0010] In one possible implementation of the first aspect, the step of preprocessing the multimodal TCM prescription dataset to obtain a preprocessed multimodal TCM prescription dataset includes:

[0011] The TCM medical record text data is cleaned to remove unstructured noise characters, resulting in the cleaned TCM medical record text data.

[0012] Image denoising processing is performed on the tongue image data and the pulse diagnosis image data to eliminate uneven lighting and blurred areas of the image, resulting in denoised tongue image data and pulse diagnosis image data.

[0013] The speech description data is subjected to speech noise reduction processing to separate environmental noise from effective speech content, and the speech description data after noise reduction is obtained.

[0014] The cleaned TCM medical record text data is processed by word vectorization and generated into standardized medical record text data through L2 normalization. The denoised tongue image data and pulse diagnosis image data are generated into standardized tongue image data and standardized pulse diagnosis image data through global mean pooling and Z-Score normalization. The denoised speech description data is generated into standardized speech data through Mel spectrum feature extraction and amplitude normalization.

[0015] The standardized medical record text data, the standardized tongue image data, the standardized pulse diagnosis image data, and the standardized voice data are integrated into a preprocessed multimodal TCM prescription data set.

[0016] In one possible implementation of the first aspect, the step of performing feature extraction on the preprocessed multimodal TCM prescription data set to generate a text semantic feature set, a tongue image feature set, and a pulse diagnosis image feature set includes:

[0017] The standardized speech data is converted into standardized converted text data. A pre-trained text semantic feature extraction model is called to perform context semantic encoding on the standardized medical record text data and the standardized converted text data to generate a set of text semantic features for the standardized medical record text data and the standardized converted text data.

[0018] The pre-trained tongue image feature encoding model is invoked to perform hierarchical convolution processing on the standardized tongue image data to extract the tongue image feature set of the standardized tongue image data;

[0019] The pre-trained pulse diagnosis image feature encoding model is invoked to perform spatial feature extraction processing on the standardized pulse diagnosis image data, generating a pulse diagnosis image feature set of the standardized pulse diagnosis image data;

[0020] Furthermore, the text semantic feature set, tongue image feature set, and pulse diagnosis image feature set are input into a pre-trained multimodal alignment model, which maps the text semantic feature set, tongue image feature set, and pulse diagnosis image feature set to an embedding space of a unified dimension, thereby generating an aligned text semantic feature set, tongue image feature set, and pulse diagnosis image feature set.

[0021] For example, in one possible implementation of the first aspect, the step of calling a pre-trained tongue image feature encoding model to perform hierarchical convolution processing on the standardized tongue image data to extract the tongue image feature set of the standardized tongue image data includes:

[0022] The standardized tongue image data is input into the first multi-scale convolutional layer group of the tongue image feature encoding model to generate a shallow texture feature map containing the output of convolutional kernels of different sizes.

[0023] The shallow texture feature map is input into the second deep residual convolutional layer group of the tongue image feature encoding model. The edge features of the tongue coating region of different convolutional layers are aggregated by skip connection to generate a deep tongue body contour feature map.

[0024] The shallow texture feature map and the deep tongue contour feature map are spliced ​​together by channel dimension to generate a fused multi-scale tongue image feature map.

[0025] A global average pooling operation is performed on the multi-scale tongue image feature map to generate a fixed-dimensional primary tongue image feature vector;

[0026] The tongue region focusing module of the tongue image feature encoding model is invoked to generate a tongue region spatial attention weight matrix based on the primary tongue feature vector. The weight matrix is ​​then multiplied element-wise with the multi-scale tongue image feature map to obtain a weighted tongue image focusing feature map.

[0027] Max pooling is used to reduce the dimensionality of the weighted tongue image focusing feature map to generate a dimensionality-reduced intermediate feature vector of the tongue image.

[0028] The primary feature vector of the tongue image and the intermediate feature vector of the tongue image are concatenated and mapped to a preset unified feature dimension through a fully connected layer to generate a set of tongue image feature vectors corresponding to the standardized tongue image data.

[0029] For example, in one possible implementation of the first aspect, the step of invoking a pre-trained pulse image feature encoding model to perform spatial feature extraction processing on the standardized pulse image data to generate a pulse image feature set of the standardized pulse image data includes:

[0030] The standardized pulse diagnosis image data is divided into a set of pulse position region image blocks according to a nine-square grid layout. Each pulse position region image block corresponds to a different pressure level region of the cun, guan, and chi positions in traditional Chinese medicine pulse diagnosis.

[0031] Each pulse position region image block is input into the spatial convolution module of the pulse diagnosis image feature encoding model. Local features of the pulse waveform under multiple receptive fields are extracted through parallel dilated convolutional layers to generate primary pulse feature maps for each pulse position region image block.

[0032] Spatial attention calculation is performed on the primary pulse feature map to generate a heatmap of pulse intensity distribution in each pulse region. Based on the weight values ​​of the heatmap of pulse intensity distribution, feature filtering is performed on the primary pulse feature map to obtain a weighted pulse region feature map.

[0033] The pulse region feature maps of multiple pressure level image blocks in the same pulse position area are superimposed in a time-series dimension to generate a three-dimensional feature tensor of the pulse position.

[0034] Perform a three-dimensional pooling operation on the three-dimensional feature tensor of the pulse position to generate a compressed feature vector for each pulse position;

[0035] The compressed feature vectors of the cun, guan, and chi positions are spliced ​​together in anatomical order to generate a joint feature vector of the entire pulse position.

[0036] The joint feature vector of the entire pulse position is input into the fully connected layer group of the pulse diagnosis image feature encoding model, and mapped to a preset unified feature dimension through nonlinear transformation to generate the pulse diagnosis image feature set corresponding to the standardized pulse diagnosis image data.

[0037] In one possible implementation of the first aspect, the construction of a multimodal TCM prescription feature database based on the text semantic feature set, the tongue image feature set, and the pulse diagnosis image feature set includes:

[0038] Each text semantic feature in the text semantic feature set is associated with and stored with its corresponding TCM prescription identifier;

[0039] Each tongue image feature in the tongue image feature set is associated with and stored with the corresponding TCM prescription identifier;

[0040] Each pulse diagnosis image feature in the pulse diagnosis image feature set is associated with and stored with its corresponding traditional Chinese medicine prescription identifier;

[0041] The aligned text semantic feature set, tongue image feature set, and pulse diagnosis image feature set are used to generate a cross-modal feature vector set of unified dimension through a multimodal joint embedding model.

[0042] Based on the cross-modal feature vector set, a cross-modal feature index structure is constructed for the multimodal TCM prescription feature database, so that the cross-modal feature vector corresponding to each TCM prescription identifier can support multimodal joint similarity retrieval through the cross-modal feature index structure.

[0043] In one possible implementation of the first aspect, in response to the input multimodal query data, the multimodal TCM prescription feature database is invoked to perform a multimodal feature matching operation, generating and outputting a set of TCM prescription retrieval results, including:

[0044] Feature extraction is performed on the input multimodal query data to generate semantic features of the query text, tongue image features of the query, and pulse diagnosis image features of the query.

[0045] The algorithm is based on an approximate nearest neighbor search to calculate the first similarity score between the query text semantic features and each text semantic feature in the text semantic feature set of the cross-modal feature index structure in the multimodal TCM prescription feature database; the algorithm is also based on an approximate nearest neighbor search algorithm to calculate the second similarity score between the query tongue image features and each tongue image feature in the tongue image feature set of the cross-modal feature index structure in the multimodal TCM prescription feature database; and the algorithm is also based on an approximate nearest neighbor search algorithm to calculate the third similarity score between the query pulse diagnosis image features and each pulse diagnosis image feature in the pulse diagnosis image feature set of the cross-modal feature index structure in the multimodal TCM prescription feature database.

[0046] The first similarity score, the second similarity score, and the third similarity score are normalized so that they are all between 0 and 1.

[0047] Based on the weighted fusion result of the first similarity score, the second similarity score, and the third similarity score, a set of TCM prescription identifiers that meet the similarity threshold is selected from the multimodal TCM prescription feature database as the TCM prescription retrieval result set.

[0048] In one possible implementation of the first aspect, the step of performing feature extraction on the input multimodal query data to generate query text semantic features, query tongue image features, and query pulse diagnosis image features includes:

[0049] If the multimodal query data contains a text query statement, a pre-trained text semantic feature extraction model is invoked to perform context semantic encoding on the text query statement to generate the query text semantic features;

[0050] If the multimodal query data includes a tongue image query image, a pre-trained tongue image feature encoding model is called to perform hierarchical convolution processing on the tongue image query image to generate the query tongue image features;

[0051] If the multimodal query data contains a pulse diagnosis query image, a pre-trained pulse diagnosis image feature encoding model is invoked to perform spatial feature extraction processing on the pulse diagnosis query image to generate the query pulse diagnosis image features;

[0052] If the multimodal query data contains speech descriptions, the speech descriptions are converted to text, and then the text semantic feature extraction model is called to generate the semantic features of the query text.

[0053] In one possible implementation of the first aspect, the step of selecting a set of TCM prescription identifiers that meet a similarity threshold from the multimodal TCM prescription feature database as the TCM prescription retrieval result set based on the weighted fusion result of the first similarity score, the second similarity score, and the third similarity score includes:

[0054] Based on preset text weight coefficients, tongue image weight coefficients, and pulse image weight coefficients, the first similarity score, the second similarity score, and the third similarity score are weighted and summed to generate a comprehensive similarity score.

[0055] The set of TCM prescription identifiers is sorted from high to low according to the comprehensive similarity score to generate a sorted sequence of TCM prescription identifiers;

[0056] Based on a preset retrieval quantity threshold, the first N TCM prescription identifiers are selected from the sorted TCM prescription identifier sequence as the TCM prescription retrieval result set, where N is a positive integer and is determined by user configuration.

[0057] In one possible implementation of the first aspect, the method for determining the text weight coefficient, tongue image weight coefficient, and pulse image weight coefficient includes:

[0058] Obtain user feedback ratings on text features, tongue features, and pulse features from historical search records;

[0059] Calculate the average feedback score for text features, the average feedback score for tongue features, and the average feedback score for pulse features based on the feedback score data.

[0060] The average feedback scores of the text features, the average feedback scores of the tongue features, and the average feedback scores of the pulse features are normalized to generate the text weight coefficient, the tongue weight coefficient, and the pulse weight coefficient.

[0061] In one possible implementation of the first aspect, the multimodal TCM prescription feature database further includes a timeliness update mechanism, the execution steps of which include:

[0062] Regularly acquire newly added multimodal TCM prescription data, perform the data preprocessing and feature extraction on the newly added multimodal TCM prescription data, and generate new text semantic features, new tongue image features and new pulse diagnosis image features;

[0063] After associating the newly added text semantic features, the newly added tongue image features, and the newly added pulse diagnosis image features with the newly added TCM prescription identifier, they are written into a temporary incremental repository.

[0064] At preset time intervals, data from the temporary incremental repository is batch-merged into the multimodal TCM prescription feature database, and the cross-modal feature index structure is sharded and updated, with only the affected data shards being reconstructed.

[0065] In addition, the last access timestamp of each TCM prescription identifier in the multimodal TCM prescription feature database is monitored;

[0066] If the difference between the last access timestamp and the current timestamp of any TCM prescription identifier exceeds the preset expiration time threshold, then the TCM prescription identifier and its associated text semantic features, tongue image features, and pulse diagnosis image features are marked as expired data.

[0067] The expired data is migrated to an offline repository, and the index entries corresponding to the expired data are deleted from the multimodal TCM prescription feature database.

[0068] In another aspect, embodiments of the present invention also provide a TCM prescription retrieval system based on multimodal TCM prescription data, including a processor and a machine-readable storage medium connected to the processor. The machine-readable storage medium is used to store programs, instructions, or code, and the processor is used to execute the programs, instructions, or code in the machine-readable storage medium to implement the above-described method.

[0069] Based on the above, this invention integrates TCM medical record text data, tongue image data, pulse diagnosis image data, and voice description data to form a multimodal TCM prescription data set. After data preprocessing and feature extraction, a multimodal TCM prescription feature database is constructed. The input multimodal query data is then subjected to multimodal feature matching to output a retrieval result set. This significantly improves the comprehensiveness, accuracy, and efficiency of TCM prescription retrieval. It breaks through the limitations of traditional retrieval methods that rely solely on single text or a few features. By comprehensively utilizing multimodal data to reflect the overall information of TCM diagnosis, the retrieval can more accurately capture the correlation between patient condition characteristics and prescriptions, providing more reliable and comprehensive prescription retrieval support for TCM clinical practice, teaching, and research. Attached Figure Description

[0070] Figure 1 This is a schematic diagram of the execution flow of the TCM prescription retrieval method based on multimodal TCM prescription data provided in an embodiment of the present invention.

[0071] Figure 2 This is a schematic diagram of exemplary hardware and software components of a TCM prescription retrieval system based on multimodal TCM prescription data provided in an embodiment of the present invention. Detailed Implementation

[0072] The present invention will now be described in detail with reference to the accompanying drawings. Figure 1 This is a flowchart illustrating a TCM prescription retrieval method based on multimodal TCM prescription data according to an embodiment of the present invention. The following is a detailed description of this TCM prescription retrieval method based on multimodal TCM prescription data.

[0073] Step S110: Obtain a multimodal TCM prescription data set, which includes TCM medical record text data, tongue image data, pulse diagnosis image data, and voice description data.

[0074] In this embodiment, the TCM prescription data set includes TCM medical record text data collected from the electronic medical record systems of major TCM medical institutions. These systems store a large amount of patient medical record information, covering symptom presentation, disease diagnosis, medication records, etc. Tongue image data is acquired using specialized tongue image acquisition equipment, which can clearly capture the patient's tongue features, such as tongue color and coating, under specific lighting and shooting parameters. Pulse diagnosis image data is acquired through a high-precision pulse sensor, which can record pulse information at different locations and pressure levels. Voice description data comes from recordings of doctor-patient consultations, including detailed descriptions of the patient's symptoms and the doctor's inquiries and judgments.

[0075] For example, in various TCM medical institutions, electronic medical record systems generate a large amount of new TCM medical record text data every day, covering various symptoms and patients of different ages. Simultaneously, tongue imaging devices and pulse sensors equipped in TCM medical institutions collect relevant data from patients who come for consultation. Furthermore, the voice descriptions during the consultation process are recorded using audio recording devices. All of this data collectively constitutes a multimodal TCM prescription data set.

[0076] It is worth noting that all data, including TCM medical record text data, tongue image data, pulse diagnosis image data, and voice description data, must strictly comply with relevant laws, regulations, and ethical guidelines during collection and subsequent use. The collection of TCM medical record text data must adhere to medical data protection regulations to ensure full protection of patient privacy. For example, when obtaining TCM medical record text data from a hospital's electronic medical record system, dual authorization from both the hospital and the patient is required. Strict anonymization processing must be performed on the patient's sensitive personal information, such as name and ID number, retaining only non-sensitive information related to diagnosis and symptom description.

[0077] When collecting tongue and pulse image data, the use of specialized equipment must comply with relevant medical device regulations to ensure the safety and accuracy of the equipment. Simultaneously, the purpose and intended use of the data must be fully explained to the patient during the collection process, and their voluntary consent obtained. The collection of voice description data must also be conducted legally and compliantly, avoiding unauthorized recording of patient-doctor conversations. Furthermore, the collected voice data must be strictly stored and managed to prevent data leakage.

[0078] Step S120: Perform data preprocessing on the multimodal TCM prescription data set to obtain the preprocessed multimodal TCM prescription data set.

[0079] Because the original multimodal TCM prescription data set has many problems, such as the TCM medical record text data containing a large number of unstructured noise characters, tongue image data and pulse diagnosis image data having uneven lighting and blurry images, and voice description data mixed with environmental noise, data preprocessing is required to improve data quality. For details, please refer to the following examples.

[0080] Step S121: Perform text data cleaning processing on the TCM medical record text data to remove unstructured noise characters and obtain the cleaned TCM medical record text data.

[0081] In traditional Chinese medicine (TCM) medical record text data, unstructured noise characters can interfere with subsequent text analysis. These unstructured noise characters may include punctuation marks, special symbols, garbled characters, etc. To remove these noise characters, regular expression matching can be used. Let the original TCM medical record text data be T, and the regular expression pattern be R. By performing a matching operation, characters in T that match the pattern R are removed, resulting in the cleaned TCM medical record text data T'.

[0082] For example, if T is "Patient @ has # fever symptoms!", and R is set to match all special characters, after processing, T' will be "Patient has fever symptoms". This cleaning operation can make the text data cleaner, which is easier for subsequent word vectorization and other processing.

[0083] Step S122: Perform image denoising processing on the tongue image data and the pulse diagnosis image data to eliminate uneven lighting and blurred areas of the image, and obtain the denoised tongue image data and pulse diagnosis image data.

[0084] For tongue and pulse diagnosis image data, uneven lighting and image blurring can affect the extraction of image features. Various methods can be used for image denoising, such as filtering-based methods. Let the original tongue image data be I_t and the pulse diagnosis image data be I_m.

[0085] First, to address the issue of uneven illumination, histogram equalization can be used. Histogram equalization is applied to I_t and I_m respectively, adjusting the brightness and contrast of the image to make the overall illumination distribution more uniform. Let the tongue image data after histogram equalization be I_t1, and the pulse diagnosis image data be I_m1.

[0086] Then, for blurred areas in the image, Gaussian filtering can be used for denoising. Taking tongue image data as an example, a Gaussian filter is applied to I_t1, and the filter parameters are adjusted according to the specific situation of the image to obtain the denoised tongue image data I_t2. The same method is applied to the pulse diagnosis image data I_m1 to obtain the denoised pulse diagnosis image data I_m2.

[0087] For example, in a tongue image, due to lighting conditions, some areas of the tongue are darker than others. After histogram equalization, the brightness of the entire tongue image becomes more uniform. Further Gaussian filtering removes blur and noise, making the texture and features of the tongue clearer and more discernible.

[0088] Step S123: Perform speech noise reduction processing on the speech description data to separate environmental noise from effective speech content, and obtain the noise-reduced speech description data.

[0089] Speech description data often contains environmental noise, which can affect subsequent speech analysis and conversion. To separate environmental noise from effective speech content, spectral subtraction can be used. Let the original speech description data be S, and its spectrum be P_S.

[0090] First, a spectral analysis is performed on S to obtain its spectrum P_S. Then, the spectrum of environmental noise P_N is estimated, which can be obtained by performing spectral statistics on the silent segments of the speech signal. Next, P_N is subtracted from P_S to obtain the noise-removed spectrum P_S'. Finally, P_S' is converted into the denoised speech description data S' using an inverse Fourier transform.

[0091] For example, during a medical consultation, the surrounding environment may be noisy, causing the recorded speech data to contain environmental noise. Spectral subtraction can effectively separate environmental noise from the speech signal, resulting in clear and valid speech content.

[0092] Step S124: The cleaned TCM medical record text data is processed by word vectorization and standardized medical record text data is generated by L2 normalization. The denoised tongue image data and pulse diagnosis image data are standardized by global mean pooling and Z-Score normalization. The denoised speech description data is standardized by Mel spectrum feature extraction and amplitude normalization.

[0093] For the cleaned TCM medical record text data T', word vectorization is performed. A pre-trained word vector model, such as the Word2Vec model, can be used. Each word in T' is mapped to a corresponding word vector, resulting in a word vector sequence V. Let the dimension of each word vector in V be d.

[0094] Then, L2 normalization is performed on the word vector sequence V. For each word vector v in V, its L2 norm ||v|| is calculated, and then v is divided by ||v|| to obtain the normalized word vector v'. All normalized word vectors are combined to obtain the standardized medical record text data T''.

[0095] For the denoised tongue image data I_t2 and pulse diagnosis image data I_m2, a global mean pooling operation is first performed. Taking the tongue image data as an example, let the size of I_t2 be h×w×c (h is the height, w is the width, and c is the number of channels). Mean pooling is performed on I_t2 in both the height and width directions to obtain a fixed-length feature vector F_t1. The same method is applied to the pulse diagnosis image data I_m2 to obtain the feature vector F_m1.

[0096] Next, Z-score standardization is performed on F_t1 and F_m1. Let the mean of F_t1 be μ_t and the standard deviation be σ_t. The standardized tongue image feature vector F_t2 is obtained using the formula F_t2=(F_t1-μ_t) / σ_t. Similarly, F_m1 is processed to obtain the standardized pulse diagnosis image feature vector F_m2.

[0097] For the denoised speech description data S', Mel-spectral feature extraction is performed. S' is divided into multiple short time frames, and a Fast Fourier Transform is performed on each frame. Then, the spectrum is converted into a Mel spectrum. Let M be the resulting Mel-spectral matrix.

[0098] Then, amplitude normalization is performed on M. The maximum and minimum values ​​of each element in M ​​are calculated, and the elements in M ​​are mapped to the interval [0, 1] through a linear transformation to obtain the standardized speech data S''.

[0099] Step S125: Integrate the standardized medical record text data, the standardized tongue image data, the standardized pulse diagnosis image data, and the standardized voice data into a preprocessed multimodal TCM prescription data set.

[0100] In this embodiment, the standardized medical record text data T'', standardized tongue image data F_t2, standardized pulse diagnosis image data F_m2, and standardized speech data S'' obtained from the previous steps are integrated together. These data from different modalities can be concatenated in a predetermined order to form a new multimodal data vector D. This multimodal data vector D is the preprocessed multimodal TCM prescription data set, containing the feature information of each modality.

[0101] Step S130: Perform feature extraction on the preprocessed multimodal TCM prescription data set to generate a text semantic feature set, a tongue image feature set, and a pulse diagnosis image feature set.

[0102] After data preprocessing, valuable features need to be extracted from the preprocessed multimodal TCM prescription dataset to facilitate the subsequent construction of a multimodal TCM prescription feature database. Different feature extraction methods are used for different modalities of data.

[0103] Step S131: Perform speech conversion on the standardized speech data to obtain standardized converted text data. Call the pre-trained text semantic feature extraction model to perform context semantic encoding processing on the standardized medical record text data and the standardized converted text data to generate a set of text semantic features for the standardized medical record text data and the standardized converted text data.

[0104] For standardized speech data S'', speech recognition technology is used to convert it into text data, resulting in standardized converted text data T_s. Open-source speech recognition tools, such as Google Cloud Speech-to-Text or speech recognition APIs, can be used.

[0105] Then, a pre-trained text semantic feature extraction model, such as the BERT model, is invoked. The standardized medical record text data T'' and the standardized transformed text data T_s are input into the BERT model respectively. The BERT model performs contextual semantic encoding on the input text, mapping each word in the text to a high-dimensional semantic vector space.

[0106] Let the output dimension of the BERT model be D_b. For standardized medical record text data T'', we obtain the corresponding text semantic feature vector sequence V_t. For standardized transformed text data T_s, we obtain the corresponding text semantic feature vector sequence V_s. Combining V_t and V_s yields the text semantic feature set V_text.

[0107] Step S132: Call the pre-trained tongue image feature encoding model to perform hierarchical convolution processing on the standardized tongue image data, and extract the tongue image feature set of the standardized tongue image data.

[0108] In this embodiment, the pre-trained tongue image feature encoding model has a specific structure, which is used to perform hierarchical convolution processing on the standardized tongue image data F_t2 to extract its tongue image features.

[0109] Step S1321: Input the standardized tongue image data into the first multi-scale convolutional layer group of the tongue image feature encoding model to generate a shallow texture feature map containing the output of convolutional kernels of different sizes.

[0110] Let the standardized tongue image data F_t2 have dimensions h_t × w_t × c_t. The first multi-scale convolutional layer group contains multiple convolutional kernels of different sizes, such as 3×3, 5×5, etc. When F_t2 is input into the first multi-scale convolutional layer group, convolutional kernels of different sizes will perform convolution operations on the image.

[0111] Taking a 3×3 convolution kernel as an example, let the weight matrix of the convolution kernel be W_3. Performing a convolution operation on F_t2 yields a feature map F_3. Similarly, performing a convolution operation using a 5×5 convolution kernel (with a weight matrix of W_5) yields a feature map F_5. Combining these feature maps output by different convolution kernels yields a shallow texture feature map F_texture containing the outputs of convolution kernels of different sizes.

[0112] Step S1322: Input the shallow texture feature map into the second deep residual convolutional layer group of the tongue image feature encoding model, and aggregate the edge features of the tongue coating region of different convolutional layers through skip connection to generate a deep tongue contour feature map.

[0113] The second depth residual convolutional layer group consists of multiple residual blocks. Each residual block contains a convolutional layer, a batch normalization layer, and an activation function layer. When the shallow texture feature map F_texture is input into the second depth residual convolutional layer group, the residual blocks perform further convolutional processing on it.

[0114] Within the residual blocks, skip connections are used to add the input feature map to the convolutional feature map. This effectively aggregates feature information from different convolutional layers, especially the edge features of the tongue coating region. After processing through multiple residual blocks, the deep tongue contour feature map F_contour is obtained.

[0115] Step S1323: Perform channel dimension splicing on the shallow texture feature map and the deep tongue contour feature map to generate a fused multi-scale tongue image feature map.

[0116] The shallow texture feature map F_texture and the deep tongue contour feature map F_contour are concatenated along the channel dimension. Let the number of channels in F_texture be c_texture and the number of channels in F_contour be c_contour. Then, the number of channels in the concatenated and fused multi-scale tongue image feature map F_fused is c_texture + c_contour. This concatenation method integrates shallow texture features and deep contour features, enriching the expression of tongue image features.

[0117] Step S1324: Perform global average pooling on the multi-scale tongue image feature map to generate a fixed-dimensional primary tongue image feature vector.

[0118] Global average pooling is performed on the fused multi-scale tongue image feature map F_fused. Average pooling is applied to F_fused in both the height and width directions, averaging the feature values ​​of each channel to obtain a fixed-length vector. Let the dimensions of F_fused be h_f × w_f × c_fused. After global average pooling, the primary tongue image feature vector F_primary is obtained, with dimension c_fused.

[0119] Step S1325: Call the tongue region focusing module of the tongue image feature coding model, generate a tongue region spatial attention weight matrix based on the primary feature vector of the tongue image, and multiply the weight matrix element-wise with the multi-scale tongue image feature map to obtain a weighted tongue image focusing feature map.

[0120] The tongue image region focusing module receives the primary tongue image feature vector F_primary as input. This module generates a spatial attention weight matrix W for the tongue region through a series of fully connected layers and activation functions. The dimensions of W are the same as the height and width of the multi-scale tongue image feature map F_fused.

[0121] Then, the weight matrix W is multiplied element-wise with the multi-scale tongue image feature map F_fused. Let the element in the i-th row, j-th column, k-th channel of F_fused be F_fused(i, j, k), and the element in the i-th row, j-th column of W be W(i, j). Then, the element in the i-th row, j-th column, k-th channel of the weighted tongue image focused feature map F_focused is F_focused(i, j, k) = F_fused(i, j, k) × W(i, j). In this way, the important features of the tongue body region are highlighted.

[0122] Step S1326: Perform max pooling dimensionality reduction on the weighted tongue image focusing feature map to generate the dimensionality-reduced tongue image intermediate feature vector.

[0123] Max pooling is used to reduce the dimensionality of the weighted tongue image focused feature map F_focused. A pooling window of fixed size slides along the height and width directions, and the maximum value within each window is taken as the output. Let the size of the pooling window be p_h×p_w. After max pooling, the dimensionality-reduced tongue image intermediate feature vector F_intermediate is obtained, whose dimension is smaller than the number of channels in F_focused.

[0124] Step S1327: Concatenate the primary feature vector of the tongue image with the intermediate feature vector of the tongue image, and map them to a preset unified feature dimension through a fully connected layer to generate a set of tongue image feature vectors corresponding to the standardized tongue image data.

[0125] The primary tongue image feature vector F_primary and the intermediate tongue image feature vector F_intermediate are concatenated to obtain the concatenated vector F_combined. Then, F_combined is input into a fully connected layer. The weight matrix of the fully connected layer is W_fc, and the bias vector is b_fc. Through the linear transformation F_output = W_fc × F_combined + b_fc, F_combined is mapped to a preset uniform feature dimension. The feature vectors corresponding to all processed and standardized tongue image data are combined to obtain the tongue image feature set V_tongue.

[0126] Step S133: Call the pre-trained pulse diagnosis image feature encoding model to perform spatial feature extraction processing on the standardized pulse diagnosis image data, and generate a pulse diagnosis image feature set of the standardized pulse diagnosis image data.

[0127] A pre-trained pulse diagnosis image feature encoding model is used to extract spatial features from standardized pulse diagnosis image data F_m2 to obtain feature information of the pulse diagnosis image.

[0128] Step S1331: Divide the standardized pulse diagnosis image data into a set of pulse position region image blocks according to a nine-square grid layout, wherein each pulse position region image block corresponds to a different pressure level region of the cun, guan, and chi positions in traditional Chinese medicine pulse diagnosis.

[0129] Let the standardized pulse diagnosis image data F_m2 have dimensions of h_m × w_m × c_m. Divide F_m2 into a 3x3 grid to obtain 9 pulse position image blocks. Each pulse position image block corresponds to a different pressure level region at the cun, guan, and chi positions in traditional Chinese medicine pulse diagnosis. Let the i-th pulse position image block be B_i, with dimensions of h_b × w_b × c_m (h_b and w_b are the height and width of the segmented image block).

[0130] Step S1332: Input each pulse position region image block into the spatial convolution module of the pulse diagnosis image feature encoding model, extract local features of pulse waveform under multiple receptive fields through parallel dilated convolution layers, and generate primary pulse feature maps for each pulse position region image block.

[0131] The spatial convolution module contains parallel dilated convolutional layers. When each pulse region image block B_i is input into the spatial convolution module, the parallel dilated convolutional layers perform convolution operations on it. The different dilation rates of the dilated convolutional layers allow for the extraction of local pulse waveform features from different receptive fields.

[0132] Taking a dilated convolutional layer as an example, let the dilation rate be r and the weight matrix of the convolutional kernel be W_r. Perform a convolution operation on B_i to obtain a feature map F_r_i. Combine the feature maps output by dilated convolutional layers with different dilation rates to obtain the primary pulse feature map F_primary_m of each pulse position region image block.

[0133] Step S1333: Perform spatial attention calculation on the primary pulse feature map to generate a pulse intensity distribution heatmap for each pulse location region, and perform feature filtering on the primary pulse feature map according to the weight values ​​of the pulse intensity distribution heatmap to obtain a weighted pulse region feature map.

[0134] Spatial attention is calculated on the primary pulse feature map F_primary_m. First, F_primary_m is processed through a small convolutional neural network to obtain the spatial attention weight matrix W_m. The dimensions of W_m are the same as the height and width of F_primary_m.

[0135] Then, a heatmap of pulse intensity distribution for each pulse region is generated based on W_m. Elements in W_m are mapped to a color space, with darker colors indicating greater pulse intensity in that region. Next, feature filtering is performed on the primary pulse feature map F_primary_m based on the weight values ​​of the pulse intensity distribution heatmap. Elements with lower weight values ​​in F_primary_m are set to 0, resulting in the weighted pulse region feature map F_weighted_m.

[0136] Step S1334: Overlay the weighted pulse region feature maps of multiple pressure level image blocks in the same pulse position region in a temporal dimension to generate a three-dimensional feature tensor of the pulse position.

[0137] For the weighted pulse region feature maps of multiple pressure level image patches within the same pulse location region, they are superimposed in the temporal dimension. Let there be n weighted pulse region feature maps of the same pulse location region, denoted as F_weighted_m_1, F_weighted_m_2, ..., F_weighted_m_n, each with dimensions h_m_b × w_m_b × c_m_b (where h_m_b and w_m_b are the height and width of the image patch, and c_m_b is the number of channels). These feature maps are superimposed in the temporal dimension, i.e., arranged sequentially, forming a three-dimensional tensor structure, generating a three-dimensional pulse location feature tensor T_m. This three-dimensional pulse location feature tensor T_m contains temporal information of pulse features at different pressure levels within the same pulse location region, providing richer data for subsequent feature compression and integration.

[0138] Step S1335: Perform a three-dimensional pooling operation on the three-dimensional feature tensor of the pulse position to generate a compressed feature vector for each pulse position.

[0139] A 3D pooling operation is performed on the 3D feature tensor T_m of the pulse position. 3D pooling is similar to 2D pooling, but it's performed in three dimensions. Either 3D max pooling or 3D average pooling can be used; here, we'll use 3D max pooling as an example. A 3D pooling window is defined with a size of p_h_m × p_w_m × p_d_m (where p_h_m and p_w_m are the pooling window sizes in the height and width directions, respectively, and p_d_m is the pooling window size in the depth direction, i.e., the time dimension). This pooling window is slid across the 3D feature tensor T_m of the pulse position, and the maximum value within the window is taken as the output each time. Through this operation, the 3D feature tensor T_m of the pulse position is reduced in dimensionality, decreasing the dimensionality and quantity of data, generating a compressed feature vector V_m_compressed for each pulse position. This compressed feature vector retains the main information of the pulse position features while reducing data complexity.

[0140] Step S1336: The compressed feature vectors of the cun, guan, and chi positions are spliced ​​together in anatomical order to generate a combined feature vector of the entire pulse position.

[0141] The compressed feature vectors V_m_compressed_cun, V_m_compressed_guan, and V_m_compressed_chi from the cun, guan, and chi positions are concatenated in anatomical order. This concatenation involves linking these vectors sequentially along their vector dimensions. Assuming V_m_compressed_cun has dimension d_cun, V_m_compressed_guan has dimension d_guan, and V_m_compressed_chi has dimension d_chi, the resulting combined feature vector V_m_combined will have dimensions d_cun + d_guan + d_chi. This concatenation method integrates the feature information from different pulse positions, forming a combined feature vector containing information from all pulse positions, thus providing a more comprehensive reflection of the pulse characteristics.

[0142] Step S1337: Input the joint feature vector of the entire pulse position into the fully connected layer group of the pulse diagnosis image feature coding model, and map it to a preset unified feature dimension through nonlinear transformation to generate the pulse diagnosis image feature set corresponding to the standardized pulse diagnosis image data.

[0143] The combined feature vector V_m_combined for all pulse positions is input into the fully connected layer group of the pulse diagnosis image feature encoding model. The fully connected layer group contains multiple fully connected layers, each consisting of a weight matrix and a bias vector. Assuming the weight matrix of the first fully connected layer is W_m_fc_1 and the bias vector is b_m_fc_1, the input combined feature vector V_m_combined undergoes a linear transformation in this fully connected layer to obtain the intermediate vector V_m_intermediate_1, which is equal to the product of W_m_fc_1 and V_m_combined, plus b_m_fc_1. Then, a non-linear activation function, such as ReLU, is applied to V_m_intermediate_1 to obtain the activated intermediate vector V_m_activated_1. This V_m_activated_1 is then used as input to the next fully connected layer, repeating the linear transformation and non-linear activation process. After processing through multiple fully connected layers, it is finally mapped to a preset unified feature dimension. By combining the feature vectors obtained from processing all the standardized pulse diagnosis image data in this way, the pulse diagnosis image feature set V_pulse corresponding to the standardized pulse diagnosis image data is generated.

[0144] Step S134: Input the text semantic feature set, tongue image feature set and pulse diagnosis image feature set into a pre-trained multimodal alignment model, map the text semantic feature set, tongue image feature set and pulse diagnosis image feature set to a unified dimension embedding space, and generate aligned text semantic feature set, tongue image feature set and pulse diagnosis image feature set.

[0145] The pre-trained multimodal alignment model maps feature sets from different modalities—text semantic features (V_text), tongue image features (V_tongue), and pulse image features (V_pulse)—to a unified-dimensional embedding space. This multimodal alignment model can be a neural network-based model that receives these three feature sets as input. Internally, the model transforms the input feature sets through a series of fully connected layers and non-linear activation functions. For the text semantic feature set V_text, the model learns its text semantic feature distribution and maps it to a unified-dimensional embedding space, resulting in the aligned text semantic feature set V_text_aligned. Similarly, for the tongue image feature set V_tongue and the pulse image feature set V_pulse, the model processes them separately, adjusting their feature distributions to a unified-dimensional embedding space that matches the features of the other modalities, generating the aligned tongue image feature set V_tongue_aligned and the aligned pulse image feature set V_pulse_aligned. In this way, features from different modalities can be compared and analyzed in the same space.

[0146] Step S140: Construct a multimodal TCM prescription feature database based on the text semantic feature set, the tongue image feature set, and the pulse diagnosis image feature set.

[0147] To achieve efficient TCM prescription retrieval, it is necessary to construct a multimodal TCM prescription feature database based on the extracted and aligned feature set.

[0148] Step S141: Associate and store each text semantic feature in the text semantic feature set with the corresponding TCM prescription identifier.

[0149] The text semantic feature set V_text_aligned contains multiple text semantic feature vectors, each representing the semantic features of a TCM medical record text or audio description. For each text semantic feature vector, a corresponding TCM prescription identifier is associated with it. This TCM prescription identifier is unique and used to identify a specific TCM prescription. This association information can be stored in a database table, with one column storing the text semantic feature vectors and another column storing the corresponding TCM prescription identifiers. In this way, during subsequent retrieval, the corresponding TCM prescription can be quickly located based on the text semantic features.

[0150] Step S142: Associate and store each tongue image feature in the tongue image feature set with the corresponding TCM prescription identifier.

[0151] Each tongue image feature vector in the tongue image feature set V_tongue_aligned is also associated with a corresponding Traditional Chinese Medicine (TCM) prescription identifier. Similar to the method of storing text semantic features, the tongue image feature vectors and their corresponding TCM prescription identifiers are stored in a database table. This associated storage method enables accurate retrieval of the corresponding TCM prescription when performing tongue image feature matching.

[0152] Step S143: Associate and store each pulse diagnosis image feature in the pulse diagnosis image feature set with the corresponding TCM prescription identifier.

[0153] Each pulse diagnosis image feature vector in the pulse diagnosis image feature set V_pulse_aligned is also associated with a corresponding traditional Chinese medicine (TCM) prescription identifier. This association information is stored in a database table, facilitating the rapid retrieval of the corresponding TCM prescription during pulse diagnosis image feature matching. In this way, features from different modalities are effectively associated with TCM prescriptions, laying the foundation for multimodal feature matching and retrieval.

[0154] Step S144: The aligned text semantic feature set, tongue image feature set, and pulse diagnosis image feature set are used to generate a cross-modal feature vector set of unified dimension through a multimodal joint embedding model.

[0155] The multimodal joint embedding model receives aligned text semantic feature set V_text_aligned, tongue image feature set V_tongue_aligned, and pulse diagnosis image feature set V_pulse_aligned as input. The purpose of this multimodal joint embedding model is to fuse the feature information from these three different modalities to generate a unified-dimensional cross-modal feature vector set. Internally, the multimodal joint embedding model contains multiple fully connected layers and non-linear activation functions. First, the three input feature sets are concatenated, linking them along the vector dimension to obtain a concatenated feature vector set. Then, this concatenated feature vector set is input into a fully connected layer for linear transformation, followed by a non-linear transformation through a non-linear activation function. After multiple such processes, it is mapped to a unified-dimensional space, generating the cross-modal feature vector set V_cross_modal. This cross-modal feature vector set integrates feature information from text, tongue image, and pulse diagnosis, more comprehensively reflecting the characteristics of traditional Chinese medicine prescriptions.

[0156] Step S145: Construct a cross-modal feature index structure for the multimodal TCM prescription feature database based on the cross-modal feature vector set, so that the cross-modal feature vector corresponding to each TCM prescription identifier can support multimodal joint similarity retrieval through the cross-modal feature index structure.

[0157] A cross-modal feature index structure is constructed based on the cross-modal feature vector set V_cross_modal. Index structures such as KD-trees and ball trees can be used; here, we take KD-trees as an example. A KD-tree is a data structure used for efficient high-dimensional space searching. First, each cross-modal feature vector in the V_cross_modal set is associated with its corresponding TCM prescription identifier. Then, a KD-tree is constructed using these cross-modal feature vectors as nodes. During the KD-tree construction process, appropriate partitioning dimensions and partitioning points are selected to divide the high-dimensional space into different regions. Thus, when performing multimodal joint similarity retrieval, the KD-tree structure can be used to quickly search for cross-modal feature vectors similar to the query feature vector in the high-dimensional space, thereby finding the corresponding TCM prescription identifier. This cross-modal feature index structure improves the efficiency of multimodal feature matching retrieval, enabling the system to quickly and accurately find TCM prescriptions that meet the criteria.

[0158] Step S150: In response to the input multimodal query data, the multimodal TCM prescription feature database is invoked to perform multimodal feature matching operation, generate a set of TCM prescription retrieval results and output them.

[0159] When receiving input multimodal query data, it is necessary to call the multimodal TCM prescription feature database to perform multimodal feature matching operations in order to generate and output a set of TCM prescription retrieval results.

[0160] It is worth noting that in this step, during the recommendation decision-making process, certain TCM prescriptions should not be prioritized or excluded based on any non-medical factors of the patient, such as age, gender, race, or social status. When filtering the set of TCM prescription identifiers that meet the similarity threshold, the sorting and selection must be strictly based on the comprehensive similarity score. Each TCM prescription should be given a fair chance to compete, considering only its similarity to the query data, without considering other irrelevant factors. In other words, when outputting the TCM prescription search results to the user, it is necessary to ensure that the presentation of the results is objective and fair, without any biased recommendations or guidance.

[0161] Step S151: Perform feature extraction on the input multimodal query data to generate semantic features of the query text, features of the query tongue image, and features of the query pulse diagnosis image.

[0162] The input multimodal query data may include data of different modalities, such as text query statements, tongue image query images, pulse diagnosis query images, and voice descriptions. Appropriate feature extraction methods are employed for different modalities of data.

[0163] Step S1511: If the multimodal query data contains a text query statement, call the pre-trained text semantic feature extraction model to perform context semantic encoding processing on the text query statement to generate the query text semantic features.

[0164] If the multimodal query data contains text query statements, these statements are input into a pre-trained text semantic feature extraction model, such as the BERT model. The BERT model performs contextual semantic encoding on the text query statement, mapping each word in the text to a high-dimensional semantic vector space. After processing by the model, a query text semantic feature vector V_query_text is generated. This query text semantic feature vector represents the semantic features of the text query statement and is used for subsequent feature matching operations.

[0165] Step S1512: If the multimodal query data includes a tongue image query image, call the pre-trained tongue image feature encoding model to perform hierarchical convolution processing on the tongue image query image to generate the query tongue image features.

[0166] When the multimodal query data includes tongue image query images, the tongue image query images are input into a pre-trained tongue image feature encoding model. This model performs hierarchical convolution processing on the tongue image query images, similar to the processing of standardized tongue image data in step S132. First, the tongue image query images are input into the first multi-scale convolutional layer group to generate shallow texture feature maps containing the outputs of convolutional kernels of different sizes. Next, the shallow texture feature maps are input into the second deep residual convolutional layer group, aggregating the edge features of the tongue coating region from different convolutional layers through skip connections to generate a deep tongue contour feature map. Then, the shallow texture feature map and the deep tongue contour feature map are concatenated along their channel dimensions to generate a fused multi-scale tongue image feature map. Finally, global average pooling is performed on the multi-scale tongue image feature map to generate a fixed-dimensional primary tongue image feature vector. Next, the tongue image region focusing module is invoked to generate a spatial attention weight matrix for the tongue region based on the primary tongue image feature vector. This weight matrix is ​​then element-wise multiplied with the multi-scale tongue image feature map to obtain a weighted tongue image focusing feature map. Max pooling is then applied to the weighted tongue image focusing feature map for dimensionality reduction, generating a dimensionality-reduced intermediate tongue image feature vector. Finally, the primary and intermediate tongue image feature vectors are concatenated and mapped to a preset unified feature dimension through a fully connected layer to generate the query tongue image feature vector V_query_tongue.

[0167] Step S1513: If the multimodal query data contains a pulse diagnosis query image, call the pre-trained pulse diagnosis image feature encoding model to perform spatial feature extraction processing on the pulse diagnosis query image to generate the query pulse diagnosis image features.

[0168] If the multimodal query data includes pulse diagnosis query images, the pulse diagnosis query images are input into a pre-trained pulse diagnosis image feature encoding model. This model performs spatial feature extraction on the pulse diagnosis query images, a process similar to the processing of standardized pulse diagnosis image data in step S133. First, the pulse diagnosis query images are divided into a set of pulse position region image blocks according to a nine-square grid layout. Then, each pulse position region image block is input into a spatial convolution module, and local features of the pulse waveform under multiple receptive fields are extracted through parallel dilated convolutional layers to generate primary pulse feature maps for each pulse position region image block. Next, spatial attention calculation is performed on the primary pulse feature maps to generate a pulse intensity distribution heatmap for each pulse position region. Features of the primary pulse feature maps are then filtered based on the weight values ​​of the pulse intensity distribution heatmap to obtain weighted pulse position region feature maps. Afterward, the weighted pulse position region feature maps of multiple pressure level image blocks in the same pulse position region are time-series superimposed to generate a three-dimensional pulse position feature tensor. Three-dimensional pooling is performed on the three-dimensional pulse position feature tensor to generate compressed feature vectors for each pulse position. The compressed feature vectors from the cun, guan, and chi positions are concatenated in anatomical order to generate a joint feature vector for the entire pulse position. Finally, the joint feature vector is input into a fully connected layer group and mapped to a preset unified feature dimension through nonlinear transformation to generate the query pulse diagnosis image feature vector V_query_pulse.

[0169] Step S1514: If the multimodal query data contains speech description content, perform speech-to-text processing on the speech description content, and then call the text semantic feature extraction model to generate the query text semantic features.

[0170] If the multimodal query data contains spoken descriptions, the spoken descriptions are first converted into text data using speech recognition technology. Open-source speech recognition tools, such as Google Cloud Speech-to-Text or speech recognition APIs, can be used. After obtaining the converted text data, it is input into a pre-trained text semantic feature extraction model, such as the BERT model. The model performs contextual semantic encoding on the text data to generate a query text semantic feature vector V_query_text, in the same way as processing the text query statement in step S1511.

[0171] Step S152: Calculate the first similarity score between the query text semantic features and each text semantic feature in the text semantic feature set of the cross-modal feature index structure in the multimodal TCM prescription feature database based on the approximate nearest neighbor search algorithm; calculate the second similarity score between the query tongue image features and each tongue image feature in the tongue image feature set of the cross-modal feature index structure in the multimodal TCM prescription feature database; and calculate the third similarity score between the query pulse diagnosis image features and each pulse diagnosis image feature in the pulse diagnosis image feature set of the cross-modal feature index structure in the multimodal TCM prescription feature database.

[0172] Approximate nearest neighbor search algorithms, such as the KD-tree search algorithm, are used to calculate similarity scores for different modalities. For the query text semantic feature vector V_query_text, the KD-tree search algorithm is used to find the feature vector closest to V_query_text in the text semantic feature set V_text_aligned of the cross-modal feature index structure of the multimodal TCM prescription feature database. By calculating the distance between V_query_text and each feature vector in V_text_aligned, such as Euclidean distance or cosine distance, the distance value is converted into a similarity score, resulting in the first similarity score S_text. For the query tongue image feature vector V_query_tongue, the KD-tree search algorithm is also used to calculate the distance between V_query_tongue and each feature vector in V_tongue_aligned of the cross-modal feature index structure, and this distance is converted into a similarity score, resulting in the second similarity score S_tongue. For the pulse diagnosis image feature vector V_query_pulse, in the pulse diagnosis image feature set V_pulse_aligned of the cross-modal feature index structure, the distance between V_query_pulse and each feature vector in V_pulse_aligned is calculated using the KD tree search algorithm and converted into a similarity score, resulting in the third similarity score S_pulse.

[0173] Step S153: Normalize the first similarity score, the second similarity score, and the third similarity score respectively, so that the first similarity score, the second similarity score, and the third similarity score are all between 0 and 1.

[0174] To achieve a reasonable weighted fusion of similarity scores from different modalities, the first similarity score S_text, the second similarity score S_tongue, and the third similarity score S_pulse need to be normalized. A linear normalization method can be used. For the first similarity score S_text, let its minimum value be min_S_text and its maximum value be max_S_text. The normalized first similarity score S_text_normalized is equal to (S_text - min_S_text) divided by (max_S_text - min_S_text). The same method is applied to the second similarity score S_tongue and the third similarity score S_pulse, yielding the normalized second similarity score S_tongue_normalized and the normalized third similarity score S_pulse_normalized, respectively. After normalization, all three similarity scores are unified to between 0 and 1, facilitating subsequent weighted fusion operations.

[0175] Step S154: Based on the weighted fusion result of the first similarity score, the second similarity score and the third similarity score, select a set of TCM prescription identifiers that meet the similarity threshold from the multimodal TCM prescription feature database as the TCM prescription retrieval result set.

[0176] The first similarity score S_text_normalized, the second similarity score S_tongue_normalized, and the third similarity score S_pulse_normalized after normalization are weighted and fused. Then, a set of TCM prescription identifiers that meet the similarity threshold is selected from the multimodal TCM prescription feature database.

[0177] Step S1541: Based on the preset text weight coefficient, tongue image weight coefficient, and pulse image weight coefficient, the first similarity score, the second similarity score, and the third similarity score are weighted and summed to generate a comprehensive similarity score.

[0178] The preset weight coefficients are w_text, w_tongue, and w_pulse, with w_text + w_tongue + w_pulse equaling 1. The normalized first similarity score S_text_normalized, the second similarity score S_tongue_normalized, and the third similarity score S_pulse_normalized are weighted and summed. The comprehensive similarity score S_combined is equal to w_text multiplied by S_text_normalized plus w_tongue multiplied by S_tongue_normalized plus w_pulse multiplied by S_pulse_normalized. This weighted summation method comprehensively considers the similarity of different modal features, generating a comprehensive similarity score that fully reflects the similarity between the query data and the TCM prescription features in the database. This calculation process ensures a reasonable weight allocation for different modal features during fusion, avoiding excessive influence of a single modal feature on the result.

[0179] Step S1542: Sort the set of TCM prescription identifiers from high to low according to the comprehensive similarity score to generate a sorted sequence of TCM prescription identifiers.

[0180] After obtaining the comprehensive similarity score for each TCM prescription, it is necessary to sort all TCM prescription identifiers from highest to lowest based on their comprehensive similarity scores. A sorting algorithm, such as quicksort, can be used. First, associate each TCM prescription identifier with its corresponding comprehensive similarity score, forming a list of tuples. Then, using the comprehensive similarity score as the sorting criterion, quickly sort this list of tuples. The quicksort algorithm selects a pivot value, divides the list into two parts such that all elements in the left part are less than the pivot value, and all elements in the right part are greater than the pivot value, and then recursively sorts both parts. After sorting, a sequence of TCM prescription identifiers is obtained, arranged from highest to lowest comprehensive similarity score. The similarity between the TCM prescriptions corresponding to these identifiers and the query data is arranged from highest to lowest, facilitating subsequent selection of the most relevant TCM prescriptions based on requirements.

[0181] Step S1543: Based on a preset retrieval quantity threshold, select the first N TCM prescription identifiers from the sorted TCM prescription identifier sequence as the TCM prescription retrieval result set, where N is a positive integer and is determined by user configuration.

[0182] The preset search threshold is a positive integer N, configured by the user based on actual needs. From the sorted sequence of TCM prescription identifiers, the top N TCM prescription identifiers are selected. The TCM prescriptions corresponding to these N identifiers have the highest similarity to the query data. These N identifiers are combined to form a set of TCM prescription search results. In this way, the system can provide a certain number of the most relevant TCM prescriptions according to the user's needs, satisfying the user's requirements for TCM prescription retrieval.

[0183] Step S160: Method for determining text weight coefficient, tongue image weight coefficient and pulse image weight coefficient.

[0184] Text weight coefficients, tongue image weight coefficients, and pulse image weight coefficients play a crucial role in multimodal feature matching and fusion. Their reasonable determination can improve the accuracy and effectiveness of TCM prescription retrieval.

[0185] Step S161: Obtain user feedback rating data on text features, tongue features, and pulse features from historical search records.

[0186] User feedback ratings for text features, tongue features, and pulse features are extracted from the system's historical search records. These ratings record user satisfaction levels with TCM prescriptions provided by different modalities after each search. For example, after a search, a user might give one rating to a TCM prescription retrieved based on text features, another rating to one retrieved based on tongue features, and yet another rating to one retrieved based on pulse features. These ratings reflect users' subjective perceptions of the importance of different modalities in TCM prescription retrieval. These feedback ratings can be stored in a single data store for subsequent analysis and processing.

[0187] Step S162: Calculate the average feedback score of text features, the average feedback score of tongue features, and the average feedback score of pulse features based on the feedback score data.

[0188] The acquired feedback score data is processed to calculate the average feedback scores for text features, tongue features, and pulse features respectively. For the text feature feedback score data, all feedback scores related to the text feature are summed and then divided by the number of feedback scores to obtain the average feedback score A_text for the text feature. Similarly, the feedback score data for the tongue feature is summed and divided by the number of feedback scores to obtain the average feedback score A_tongue for the tongue feature. The same operation is performed on the feedback score data for the pulse feature to obtain the average feedback score A_pulse for the pulse feature. These average feedback scores reflect the user's overall satisfaction with different modal features in historical searches, providing a basis for determining the subsequent weight coefficients.

[0189] Step S163: Normalize the average feedback score of the text feature, the average feedback score of the tongue feature, and the average feedback score of the pulse feature to generate the text weight coefficient, the tongue weight coefficient, and the pulse weight coefficient.

[0190] To ensure that the average feedback scores A_text, A_tongue, and A_pulse of text features serve as reasonable weighting coefficients, they need to be normalized. First, find the maximum value (max_A) and minimum value (min_A) among these three average feedback scores. Then, for the average feedback score A_text of text features, the normalized text weighting coefficient w_text equals (A_text - min_A) divided by (max_A - min_A). Similarly, for the average feedback score A_tongue of tongue features, the normalized tongue weighting coefficient w_tongue equals (A_tongue - min_A) divided by (max_A - min_A). For the average feedback score A_pulse of pulse features, the normalized pulse weighting coefficient w_pulse equals (A_pulse - min_A) divided by (max_A - min_A). After normalization, the values ​​of w_text, w_tongue, and w_pulse are all between 0 and 1, and their sum is 1, satisfying the requirements for weighting coefficients. The resulting text weight coefficients, tongue image weight coefficients, and pulse image weight coefficients can reasonably allocate the importance of different modal features in multimodal feature matching and fusion based on the user's historical feedback.

[0191] Step S170: Timeliness update mechanism of multimodal TCM prescription feature database.

[0192] To ensure the accuracy and timeliness of the multimodal TCM prescription feature database, a timeliness update mechanism needs to be established to update the data and index structure in the database in a timely manner.

[0193] Step S171: Periodically acquire newly added multimodal TCM prescription data, perform the data preprocessing and feature extraction on the newly added multimodal TCM prescription data, and generate new text semantic features, new tongue image features and new pulse diagnosis image features.

[0194] New multimodal TCM prescription data is retrieved from the data source according to a preset time period, such as weekly or monthly. This new data includes TCM medical record text data, tongue image data, pulse diagnosis image data, and voice description data. The same data preprocessing and feature extraction operations as steps S120-S130 are performed on this new multimodal TCM prescription data. First, the TCM medical record text data is cleaned to remove unstructured noise characters, then word vectorization and L2 normalization are performed to generate new text semantic features. The tongue image data and pulse diagnosis image data are denoised, then normalized using global mean pooling and Z-score, and then processed by the corresponding feature extraction models to generate new tongue image features and new pulse diagnosis image features, respectively. The voice description data is denoised, then processed for speech-to-text conversion and semantic feature extraction to generate new text semantic features. Through these operations, the new multimodal TCM prescription data is converted into a feature format that can be stored and used in the database.

[0195] Step S172: After associating the newly added text semantic features, the newly added tongue image features, and the newly added pulse diagnosis image features with the newly added TCM prescription identifier, write them into the temporary incremental repository.

[0196] Each newly added text semantic feature, tongue image feature, and pulse diagnosis image feature is associated with a unique new TCM prescription identifier. This identifier uniquely identifies a new TCM prescription in the database. These associated feature data are then written to a temporary incremental repository. This temporary incremental repository is a temporary data storage area used to temporarily store the new feature data for subsequent batch merging into the multimodal TCM prescription feature database. This is done to avoid frequent direct updates to the main database and reduce the impact on database performance.

[0197] Step S173: At preset time intervals, merge the data in the temporary incremental repository into the multimodal TCM prescription feature database in batches, and update the cross-modal feature index structure by sharding, rebuilding only the affected data shards.

[0198] At preset time intervals, such as weekly or monthly, data from the temporary incremental repository is batch-merged into the multimodal TCM prescription feature database. During the merging process, newly added textual semantic features, newly added tongue image features, and newly added pulse diagnosis image features, along with their corresponding TCM prescription identifiers, are inserted into the corresponding tables in the database. Simultaneously, the cross-modal feature index structure is updated in shards. The cross-modal feature index structure is stored and managed in shards; during updates, only the data shards affected by the new data are reconstructed. For example, when the feature vector of new data falls within the range of a specific data shard, only the index structure of that data shard is reconstructed, without affecting other unaffected data shards. This reduces the workload of index updates and improves update efficiency.

[0199] Step S174: Monitor the last access timestamp of each TCM prescription identifier in the multimodal TCM prescription feature database.

[0200] In the multimodal TCM prescription feature database, the last access timestamp is recorded for each TCM prescription identifier. A timestamp field can be added to the database table, and the value of this field is updated to the current time whenever a TCM prescription is accessed. The last access timestamp of each TCM prescription identifier in the database is monitored through scheduled tasks or real-time monitoring mechanisms. This allows us to understand the usage frequency and timeliness of each TCM prescription.

[0201] Step S175: If the difference between the last access timestamp and the current timestamp of any TCM prescription identifier exceeds a preset expiration time threshold, then mark the TCM prescription identifier and its associated text semantic features, tongue image features, and pulse diagnosis image features as expired data.

[0202] A preset expiration time threshold, such as one year or six months, is established. For each monitored TCM prescription identifier, the difference between its last access timestamp and the current timestamp is calculated. If this difference exceeds the preset expiration time threshold, it indicates that the TCM prescription has not been accessed for a long time and may be outdated or no longer applicable. In this case, the TCM prescription identifier and its associated text semantic features, tongue image features, and pulse diagnosis image features are marked as expired data. A tagging field can be added to the database table, setting the tagging field of expired data to a specific value for subsequent processing.

[0203] Step S176: Migrate the expired data to an offline repository and delete the index entries corresponding to the expired data from the multimodal TCM prescription feature database.

[0204] The identifiers of expired TCM prescriptions and their associated textual semantic features, tongue image features, and pulse diagnosis image features are migrated to an offline repository. The offline repository is a storage area for long-term storage of historical data, which can utilize low-cost storage devices such as tape libraries or high-capacity hard drives. Migrating expired data to the offline repository frees up storage space in the multimodal TCM prescription feature database, improving database performance. Simultaneously, index entries corresponding to expired data are deleted from the multimodal TCM prescription feature database, and the cross-modal feature index structure is updated to ensure the accuracy and efficiency of the index structure. In this way, the timeliness and validity of the data in the multimodal TCM prescription feature database are guaranteed.

[0205] It is worth noting that when training pre-trained artificial intelligence models such as text semantic feature extraction models (e.g., BERT model), tongue image feature encoding models, and pulse diagnosis image feature encoding models, it is necessary to ensure the compliance of the training process.

[0206] Regarding data usage, training data must be from legitimate sources and undergo rigorous screening and preprocessing to remove potentially sensitive information and biased data. During model training, reasonable goals and evaluation metrics should be set to ensure that the model's performance and accuracy are based on objective medical knowledge and clinical practice. Simultaneously, the model training process must be monitored and audited to prevent the model from learning illegitimate or biased patterns.

[0207] For example, when training a tongue image feature encoding model, the tongue images in the training data should come from patients of different ages, genders, and ethnicities to ensure the model's generalization ability and fairness. When evaluating the performance of each model, multiple metrics, such as accuracy and recall, need to be used to comprehensively and objectively evaluate the model's performance.

[0208] For privacy-sensitive data, such as patients' TCM medical record text data, tongue image data, pulse diagnosis image data, and voice description data, a series of privacy protection and leakage prevention technologies should be adopted.

[0209] In terms of data storage, encryption technology is used to encrypt and store data, ensuring data security during the storage process. For example, symmetric encryption algorithms can be used to encrypt TCM medical record text data, and only authorized personnel can use the corresponding key to decrypt it. For image and audio data, more advanced encryption algorithms, such as asymmetric encryption algorithms, can be used to further enhance data security.

[0210] During data transmission, secure transmission protocols, such as SSL / TLS, are used to prevent data from being stolen or tampered with during transmission. At the same time, strict identity authentication and access control are implemented for personnel accessing the data, ensuring that only those with the appropriate permissions can access and process the data.

[0211] During data processing and analysis, differential privacy technology is employed to perturb the data, ensuring the accuracy of the analysis results while protecting patient privacy. For example, when calculating similarity scores or training models, differential privacy processing is applied to the input data, ensuring that even if the data is leaked, attackers cannot obtain the patient's specific sensitive information.

[0212] In summary, this invention, from the acquisition, preprocessing, and feature extraction of multimodal TCM prescription data to the construction and retrieval of a multimodal TCM prescription feature database, and the database's timeliness update mechanism, forms a complete TCM prescription retrieval system based on multimodal TCM prescription data. This TCM prescription retrieval system can fully utilize the information from multimodal data, improving the accuracy and efficiency of TCM prescription retrieval. Simultaneously, the data update mechanism ensures the timeliness and data quality of the database, enabling the TCM prescription retrieval system to operate continuously and stably.

[0213] Figure 2 The illustration shows exemplary hardware and software components of a TCM prescription retrieval system 100 based on multimodal TCM prescription data, which can implement the ideas of this application, according to some embodiments of this application. For example, processor 120 can be used in the TCM prescription retrieval system 100 based on multimodal TCM prescription data and to perform the functions in this application.

[0214] The TCM prescription retrieval system 100 based on multimodal TCM prescription data can be a general-purpose server or a special-purpose server; both can be used to implement the TCM prescription retrieval method based on multimodal TCM prescription data of this application. Although only one server is shown in this application, for convenience, the functions described in this application can be implemented in a distributed manner on multiple similar platforms to balance the load.

[0215] For example, a TCM prescription retrieval system 100 based on multimodal TCM prescription data may include a network port 110 connected to a network, one or more processors 120 for executing program instructions, a communication bus 130, and various forms of storage media 140, such as a disk, ROM, or RAM, or any combination thereof. Exemplarily, the TCM prescription retrieval system 100 based on multimodal TCM prescription data may also include program instructions stored in ROM, RAM, or other types of non-transitory storage media, or any combination thereof. The methods of this application can be implemented according to these program instructions. The TCM prescription retrieval system 100 based on multimodal TCM prescription data also includes an I / O interface 150 between the computer and other input / output devices.

[0216] For ease of explanation, only one processor is described in the TCM prescription retrieval system 100 based on multimodal TCM prescription data. However, it should be noted that the TCM prescription retrieval system 100 based on multimodal TCM prescription data in this application may also include multiple processors. Therefore, the steps performed by one processor as described in this application may also be performed jointly or individually by multiple processors. For example, if the processor of the TCM prescription retrieval system 100 based on multimodal TCM prescription data performs steps A and B, it should be understood that steps A and B may also be performed jointly by two different processors or individually by one processor. For example, the first processor performs step A, the second processor performs step B, or the first processor and the second processor jointly perform steps A and B.

[0217] Furthermore, this embodiment of the invention also provides a readable storage medium, wherein computer-executable instructions are preset in the readable storage medium, and when the processor executes the computer-executable instructions, the above-mentioned TCM prescription retrieval method based on multimodal TCM prescription data is implemented.

[0218] It should be noted that, in order to simplify the description of the present invention and thus help to understand one or more embodiments of the invention, multiple features may sometimes be grouped into one embodiment, drawing or description thereof in the foregoing description of the embodiments of the present invention.

Claims

1. A Chinese medicine prescription retrieval method based on multi-modal Chinese medicine prescription data, characterized in that, The method includes: A multimodal TCM prescription data set is obtained, which includes TCM medical record text data, tongue image data, pulse diagnosis image data, and voice description data; The multimodal TCM prescription dataset is preprocessed to obtain a preprocessed multimodal TCM prescription dataset. Feature extraction is performed on the preprocessed multimodal TCM prescription data set to generate a text semantic feature set, a tongue image feature set, and a pulse diagnosis image feature set; A multimodal TCM prescription feature database is constructed based on the text semantic feature set, the tongue image feature set, and the pulse diagnosis image feature set. In response to the input multimodal query data, the multimodal TCM prescription feature database is invoked to perform multimodal feature matching operations, generate a set of TCM prescription retrieval results, and output them. The preprocessed multimodal TCM prescription data set includes standardized medical record text data, standardized tongue image data, standardized pulse diagnosis image data, and standardized speech data. Feature extraction is performed on the preprocessed multimodal TCM prescription data set to generate a text semantic feature set, a tongue image feature set, and a pulse diagnosis image feature set, including: The standardized speech data is converted into standardized converted text data. A pre-trained text semantic feature extraction model is called to perform context semantic encoding on the standardized medical record text data and the standardized converted text data to generate a set of text semantic features for the standardized medical record text data and the standardized converted text data. The pre-trained tongue image feature encoding model is invoked to perform hierarchical convolution processing on the standardized tongue image data to extract the tongue image feature set of the standardized tongue image data; The pre-trained pulse diagnosis image feature encoding model is invoked to perform spatial feature extraction processing on the standardized pulse diagnosis image data, generating a pulse diagnosis image feature set of the standardized pulse diagnosis image data; Furthermore, the text semantic feature set, tongue image feature set, and pulse diagnosis image feature set are input into a pre-trained multimodal alignment model, which maps the text semantic feature set, tongue image feature set, and pulse diagnosis image feature set to an embedding space of a unified dimension, thereby generating an aligned text semantic feature set, tongue image feature set, and pulse diagnosis image feature set. The construction of a multimodal TCM prescription feature database based on the text semantic feature set, the tongue image feature set, and the pulse diagnosis image feature set includes: Each text semantic feature in the text semantic feature set is associated with and stored with its corresponding TCM prescription identifier; Each tongue image feature in the tongue image feature set is associated with and stored with the corresponding TCM prescription identifier; Each pulse diagnosis image feature in the pulse diagnosis image feature set is associated with and stored with its corresponding traditional Chinese medicine prescription identifier; The aligned text semantic feature set, tongue image feature set, and pulse diagnosis image feature set are used to generate a cross-modal feature vector set of unified dimension through a multimodal joint embedding model. Based on the cross-modal feature vector set, a cross-modal feature index structure for the multimodal TCM prescription feature database is constructed, so that the cross-modal feature vector corresponding to each TCM prescription identifier can support multimodal joint similarity retrieval through the cross-modal feature index structure; The process of calling a pre-trained tongue image feature encoding model to perform hierarchical convolution processing on the standardized tongue image data, and extracting the tongue image feature set of the standardized tongue image data, includes: The standardized tongue image data is input into the first multi-scale convolutional layer group of the tongue image feature encoding model to generate a shallow texture feature map containing the output of convolutional kernels of different sizes. The shallow texture feature map is input into the second deep residual convolutional layer group of the tongue image feature encoding model. The edge features of the tongue coating region of different convolutional layers are aggregated by skip connection to generate a deep tongue body contour feature map. The shallow texture feature map and the deep tongue contour feature map are spliced ​​together by channel dimension to generate a fused multi-scale tongue image feature map. A global average pooling operation is performed on the multi-scale tongue image feature map to generate a fixed-dimensional primary tongue image feature vector; The tongue region focusing module of the tongue image feature encoding model is invoked to generate a tongue region spatial attention weight matrix based on the primary tongue feature vector. The weight matrix is ​​then multiplied element-wise with the multi-scale tongue image feature map to obtain a weighted tongue image focusing feature map. Max pooling is used to reduce the dimensionality of the weighted tongue image focusing feature map to generate a dimensionality-reduced intermediate feature vector of the tongue image. The primary feature vector of the tongue image and the intermediate feature vector of the tongue image are concatenated and mapped to a preset unified feature dimension through a fully connected layer to generate a set of tongue image feature vectors corresponding to the standardized tongue image data. The process of calling a pre-trained pulse diagnosis image feature encoding model to perform spatial feature extraction on the standardized pulse diagnosis image data, generating a pulse diagnosis image feature set of the standardized pulse diagnosis image data, includes: The standardized pulse diagnosis image data is divided into a set of pulse position region image blocks according to a nine-square grid layout. Each pulse position region image block corresponds to a different pressure level region of the cun, guan, and chi positions in traditional Chinese medicine pulse diagnosis. Each pulse position region image block is input into the spatial convolution module of the pulse diagnosis image feature encoding model. Local features of the pulse waveform under multiple receptive fields are extracted through parallel dilated convolutional layers to generate primary pulse feature maps for each pulse position region image block. Spatial attention calculation is performed on the primary pulse feature map to generate a heatmap of pulse intensity distribution in each pulse region. Based on the weight values ​​of the heatmap of pulse intensity distribution, feature filtering is performed on the primary pulse feature map to obtain a weighted pulse region feature map. The pulse region feature maps of multiple pressure level image blocks in the same pulse position area are superimposed in a time-series dimension to generate a three-dimensional feature tensor of the pulse position. Perform a three-dimensional pooling operation on the three-dimensional feature tensor of the pulse position to generate a compressed feature vector for each pulse position; The compressed feature vectors of the cun, guan, and chi positions are spliced ​​together in anatomical order to generate a joint feature vector of the entire pulse position. The joint feature vector of the entire pulse position is input into the fully connected layer group of the pulse diagnosis image feature encoding model, and mapped to a preset unified feature dimension through nonlinear transformation to generate the pulse diagnosis image feature set corresponding to the standardized pulse diagnosis image data.

2. The method of claim 1, wherein, The step of preprocessing the multimodal TCM prescription dataset to obtain a preprocessed multimodal TCM prescription dataset includes: The TCM medical record text data is cleaned to remove unstructured noise characters, resulting in the cleaned TCM medical record text data. Image denoising processing is performed on the tongue image data and the pulse diagnosis image data to eliminate uneven lighting and blurred areas of the image, resulting in denoised tongue image data and pulse diagnosis image data. The speech description data is subjected to speech noise reduction processing to separate environmental noise from effective speech content, and the speech description data after noise reduction is obtained. The cleaned TCM medical record text data is processed by word vectorization and generated into standardized medical record text data through L2 normalization. The denoised tongue image data and pulse diagnosis image data are generated into standardized tongue image data and standardized pulse diagnosis image data through global mean pooling and Z-Score normalization. The denoised speech description data is generated into standardized speech data through Mel spectrum feature extraction and amplitude normalization. The standardized medical record text data, the standardized tongue image data, the standardized pulse diagnosis image data, and the standardized voice data are integrated into a preprocessed multimodal TCM prescription data set. 3.The method of claim 1, wherein, In response to the input multimodal query data, the system invokes the multimodal TCM prescription feature database to perform multimodal feature matching, generates a set of TCM prescription retrieval results, and outputs it, including: Feature extraction is performed on the input multimodal query data to generate semantic features of the query text, tongue image features of the query, and pulse diagnosis image features of the query. The algorithm is based on an approximate nearest neighbor search to calculate the first similarity score between the query text semantic features and each text semantic feature in the text semantic feature set of the cross-modal feature index structure in the multimodal TCM prescription feature database; the algorithm is also based on an approximate nearest neighbor search algorithm to calculate the second similarity score between the query tongue image features and each tongue image feature in the tongue image feature set of the cross-modal feature index structure in the multimodal TCM prescription feature database; and the algorithm is also based on an approximate nearest neighbor search algorithm to calculate the third similarity score between the query pulse diagnosis image features and each pulse diagnosis image feature in the pulse diagnosis image feature set of the cross-modal feature index structure in the multimodal TCM prescription feature database. The first similarity score, the second similarity score, and the third similarity score are normalized so that they are all between 0 and 1. Based on the weighted fusion result of the first similarity score, the second similarity score, and the third similarity score, a set of TCM prescription identifiers that meet the similarity threshold is selected from the multimodal TCM prescription feature database as the TCM prescription retrieval result set.

4. The method according to claim 3, wherein, The step of performing feature extraction on the input multimodal query data to generate query text semantic features, query tongue image features, and query pulse diagnosis image features includes: If the multimodal query data contains a text query statement, a pre-trained text semantic feature extraction model is invoked to perform context semantic encoding on the text query statement to generate the query text semantic features; If the multimodal query data includes a tongue image query image, a pre-trained tongue image feature encoding model is called to perform hierarchical convolution processing on the tongue image query image to generate the query tongue image features; If the multimodal query data contains a pulse diagnosis query image, a pre-trained pulse diagnosis image feature encoding model is called to perform spatial feature extraction processing on the pulse diagnosis query image to generate the query pulse diagnosis image features; If the multimodal query data contains speech descriptions, the speech descriptions are converted to text, and then the text semantic feature extraction model is called to generate the semantic features of the query text.

5. The TCM prescription retrieval method based on multimodal TCM prescription data according to claim 3, characterized in that, The weighted fusion result based on the first similarity score, the second similarity score, and the third similarity score is used to select a set of TCM prescription identifiers that meet the similarity threshold from the multimodal TCM prescription feature database as the TCM prescription retrieval result set, including: Based on preset text weight coefficients, tongue image weight coefficients, and pulse image weight coefficients, the first similarity score, the second similarity score, and the third similarity score are weighted and summed to generate a comprehensive similarity score. The set of TCM prescription identifiers is sorted from high to low according to the comprehensive similarity score to generate a sorted sequence of TCM prescription identifiers; Based on a preset retrieval quantity threshold, the first N TCM prescription identifiers are selected from the sorted TCM prescription identifier sequence as the TCM prescription retrieval result set, where N is a positive integer and is determined by user configuration.

6. The TCM prescription retrieval method based on multimodal TCM prescription data according to claim 5, characterized in that, The methods for determining the text weight coefficient, tongue image weight coefficient, and pulse image weight coefficient include: Obtain user feedback ratings on text features, tongue features, and pulse features from historical search records; Calculate the average feedback score for text features, the average feedback score for tongue features, and the average feedback score for pulse features based on the feedback score data. The average feedback scores of the text features, the average feedback scores of the tongue features, and the average feedback scores of the pulse features are normalized to generate the text weight coefficient, the tongue weight coefficient, and the pulse weight coefficient.

7. The method for retrieving traditional Chinese medicine prescriptions based on multimodal traditional Chinese medicine prescription data according to claim 1, characterized in that, The multimodal TCM prescription feature database also includes a timeliness update mechanism, the execution steps of which include: Regularly acquire newly added multimodal TCM prescription data, perform the data preprocessing and feature extraction on the newly added multimodal TCM prescription data, and generate new text semantic features, new tongue image features and new pulse diagnosis image features; After associating the newly added text semantic features, the newly added tongue image features, and the newly added pulse diagnosis image features with the newly added TCM prescription identifier, they are written into a temporary incremental repository. At preset time intervals, data from the temporary incremental repository is batch-merged into the multimodal TCM prescription feature database, and the cross-modal feature index structure is sharded and updated, with only the affected data shards being reconstructed. In addition, the last access timestamp of each TCM prescription identifier in the multimodal TCM prescription feature database is monitored; If the difference between the last access timestamp and the current timestamp of any TCM prescription identifier exceeds the preset expiration time threshold, then the TCM prescription identifier and its associated text semantic features, tongue image features, and pulse diagnosis image features are marked as expired data. The expired data is migrated to an offline repository, and the index entries corresponding to the expired data are deleted from the multimodal TCM prescription feature database.

8. A traditional Chinese medicine prescription retrieval system based on multimodal traditional Chinese medicine prescription data, characterized in that, The method includes a processor and a memory, the memory and the processor being connected. The memory is used to store programs, instructions or code, and the processor is used to execute the programs, instructions or code in the memory to implement the TCM prescription retrieval method based on multimodal TCM prescription data as described in any one of claims 1-7.

Images