Traditional Chinese medicine ancient book symptom standardization processing method based on large language model

Abstract

The present invention belongs to the field of natural language processing technology, and discloses a method for standardizing symptoms in ancient Chinese medical books based on a large language model, including: obtaining and preprocessing the text of ancient Chinese medical books, extracting the initial symptom expression; for the complex symptoms existing in the initial symptom expression, splitting them by combining a preset rule engine and a large language model to obtain a single symptom expression; calculating the multidimensional similarity of the single symptom expression based on a preset standard symptom vocabulary; weighted fusion of the multidimensional similarity to obtain a comprehensive similarity score, and determining a corresponding standard symptom from the standard symptom vocabulary based on the comprehensive similarity score, and using it as a standardized expression of the initial symptom expression. The present invention can effectively overcome the matching failure problem caused by "same symptom different names", differences in word usage habits, etc. through a multi-dimensional verification mechanism, and its accuracy and robustness far exceed the existing technology that relies only on single semantic matching.

Description

Technical Field

[0001] The present invention relates to the field of natural language processing technology, and in particular to a method for standardizing symptoms in ancient Chinese medical books based on a large language model. Background Art

[0002] Ancient Chinese medical texts represent the crystallization of thousands of years of Chinese medical wisdom, containing a vast amount of valuable knowledge on diseases, symptoms, prescriptions, and treatment concepts. To better preserve and utilize this precious cultural heritage, academia and industry have undertaken extensive digitization efforts. For example, using technologies such as optical character recognition (OCR) to convert paper texts into electronic texts for easier storage and retrieval.

[0003] However, existing technologies still face severe challenges in deeply mining and utilizing the contents of these digitized ancient books, especially in the standardized processing of the core information of "symptoms":

[0004] 1. Language barriers and inconsistent terminology: The language system of ancient Chinese texts differs significantly from modern Chinese, and their expressions retain the conciseness of classical Chinese and the limitations of their time. More critically, TCM terminology lacks a unified standard. The same symptom often has multiple names in texts from different eras and schools of thought. For example, "headache" can be referred to as "head disease" or "head wind." Conversely, the same word, "fan," can refer to different symptoms, such as "heart annoyance" or "heatiness." This phenomenon of "same symptom, different names," and "different symptoms, same name," makes it difficult for simple keyword searches or matching methods to accurately and comprehensively extract symptom information.

[0005] 2. Complex symptoms lead to coarse information extraction granularity: Ancient texts often combine multiple symptoms into a single compound word. For example, "headache and neck pain" actually encompasses two separate symptoms: "headache" and "neck pain." Existing information extraction techniques often struggle to effectively and automatically separate these complex symptoms, resulting in coarse-grained information extraction that hinders subsequent accurate analysis and knowledge association.

[0006] 3. Limitations of existing information processing methods: Currently, some technical solutions attempt to solve the problem of inconsistent terminology by building a term mapping library or knowledge graph. However, the effectiveness of these methods is highly dependent on the standardization quality of the underlying symptom entities. If the input symptoms themselves are inaccurate or incomplete, then the construction of the upper-level knowledge graph will inevitably have defects. In addition, existing semantic matching methods are mostly based on modern corpus training, and their ability to understand the special grammar and context of ancient books is insufficient, making it difficult to cope with the complexity and variability of symptom descriptions in ancient books.

[0007] Therefore, the existing technology lacks an automated method that can overcome the language barriers between ancient and modern times, intelligently process complex symptoms, and perform high-precision standardized processing of ancient symptoms from multiple dimensions. This greatly hinders the systematic exploration and modern application of valuable clinical experience in ancient Chinese medicine books. Summary of the Invention

[0008] In order to solve the problems existing in the above-mentioned prior art, the present invention provides a method for automating and standardizing the symptoms recorded in ancient Chinese medical books, which specifically includes:

[0009] The method for standardizing symptoms in ancient Chinese medical texts based on a large language model includes the following steps:

[0010] S1. Obtain and preprocess ancient Chinese medical texts to extract initial symptom descriptions;

[0011] S2. Split the complex symptoms in the initial symptom description using a combination of a preset rule engine and a large language model to obtain a single symptom description;

[0012] S3. Calculating the multidimensional similarity of the single symptom expression based on a preset standard symptom vocabulary, the multidimensional similarity comprising: structural similarity based on edit distance; direct semantic similarity obtained using a first pre-trained language model; and post-translation semantic similarity obtained by converting the single symptom expression from ancient texts to modern terms using a second pre-trained language model;

[0013] S4. Perform weighted fusion on the multi-dimensional similarities to obtain a comprehensive similarity score, and determine a corresponding standard symptom from the standard symptom vocabulary based on the comprehensive similarity score, and use it as a standardized expression of the initial symptom statement.

[0014] Preferably, the method for extracting the initial symptom statement in step S1 includes: using BIO annotation rules to perform structured annotation on the ancient Chinese medical text, and using a Chinese medicine pre-trained model trained with the annotated data to identify and extract the initial symptom statement.

[0015] Preferably, the method for calculating the direct semantic similarity includes:

[0016] Fine-tuning the language model using a dataset containing pairings of standard symptom terms with symptoms from ancient books to obtain the first pre-trained language model;

[0017] The single symptom expression and the standard symptom are converted into vector representations using the first pre-trained language model, and the cosine similarity between the two vectors is calculated.

[0018] Preferably, the method for calculating the post-translation semantic similarity includes:

[0019] Fine-tuning a neural machine translation model using a dataset containing TCM symptoms and their corresponding modern terms, and using the fine-tuned neural machine translation model to convert the single symptom description from ancient Chinese medicine terms to modern terms;

[0020] Fine-tune the language model using a dataset containing standard symptom terms, corresponding modern terms, and symptom pairs from ancient books to obtain the second pre-trained language model;

[0021] The second pre-trained language model is used to convert the single symptom expression after terminology conversion and the standard symptom into vector representations, and the cosine similarity between the two vectors is calculated.

[0022] Preferably, step S2 includes:

[0023] Building the rule engine based on a preset body part dictionary and / or TCM grammar rules to split compound symptoms composed of specific body part words;

[0024] Construct prompt word templates and use the large language model to split complex symptoms that the rule engine cannot handle.

[0025] Preferably, the method for constructing the preset standard symptom vocabulary includes: constructing a multi-level mapping system including standard words, proper names and aliases based on national standards and / or industry standard terminology sets.

[0026] Preferably, the step of pre-processing the ancient TCM text in step S1 includes: removing non-text symbols in the ancient TCM text using regular expressions, and unifying variant characters or homophones in the text based on a variant character comparison table.

[0027] Beneficial effects

[0028] 1. Significantly Improved the Accuracy and Robustness of Symptom Standardization: This method combines structural similarity based on edit distance, direct semantic similarity based on a pre-trained language model, and post-translational semantic similarity through conversion of ancient and modern terms to comprehensively evaluate symptoms from three dimensions: literal structure, ancient text context, and modern context. This multi-dimensional verification mechanism effectively overcomes matching failures caused by "different names for the same symptom" and differences in word usage, and its accuracy and robustness far exceed existing technologies that rely solely on single semantic matching.

[0029] 2. Realizes refinement and automation of symptom information extraction: By splitting complex symptoms, the present invention not only realizes the automated processing of complex symptoms, but also makes the extracted symptom information finer and more accurate, providing a solid data foundation for the subsequent construction of high-quality traditional Chinese medicine knowledge graphs, data mining and clinical research.

[0030] 3. Constructing an efficient and intelligent workflow: The present invention uses a rule engine to process common symptoms with simple structures, and uses a large language model to process complex or rare long-tail symptoms. This not only ensures the processing speed and low cost for most common situations, but also uses the powerful capabilities of the large language model to solve difficult problems, forming an intelligent workflow that is both efficient and effective. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 A schematic flow chart of a method for standardizing symptoms in ancient Chinese medical books based on a large language model provided in a preferred embodiment of the present invention;

[0032] Figure 2 This is a schematic diagram of a preset standard symptom vocabulary provided in a preferred embodiment of the present invention. DETAILED DESCRIPTION

[0033] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention.

[0034] This paper proposes a multi-level, human-computer collaborative method for standardizing symptoms from ancient Chinese medical texts. The core concept of this method is to construct an automated processing pipeline that integrates in-depth analysis of ancient texts, intelligent segmentation of complex symptoms, and multi-dimensional similarity calculation using a triple model. This approach significantly improves the accuracy, coverage, and processing efficiency of symptom extraction and standardization from ancient Chinese medical texts, providing a high-quality, standardized data foundation for the subsequent construction of Traditional Chinese Medicine knowledge graphs, clinical decision support, and medical history research.

[0035] like Figure 1 As shown, this embodiment discloses a method for standardizing symptoms in ancient Chinese medical books based on a large language model, comprising the following steps:

[0036] S1. Obtain and preprocess ancient Chinese medical texts to extract initial symptom descriptions.

[0037] The processing object of the present invention is the digitized ancient Chinese medical texts. After obtaining these texts, a series of pre-processing operations are first required to eliminate data noise and lay the foundation for subsequent accurate extraction. In a preferred embodiment, the pre-processing steps include:

[0038] S11. Use regular expressions to remove non-text symbols in the text of ancient Chinese medical books, such as page numbers and proofreading symbols added by modern editors, or noise generated during the scanning process.

[0039] S12. Unify the variant characters or borrowed characters in the text based on the variant character comparison table. This step is to standardize characters in view of the common phenomenon of variant characters and borrowed characters in ancient books in order to match the modern standard vocabulary. Among them, the variant character comparison table can be constructed by referring to authoritative materials such as the "First Batch of Variant Word Sorting Tables". Based on the variant character comparison table, the variant characters or borrowed characters in the text are uniformly converted into modern standard Chinese characters. For example, "脈" and "証" are unified into "脉" and "证". This step is crucial for ensuring the consistency of subsequent model processing.

[0040] To achieve high-precision automatic extraction, in some preferred embodiments, a deep learning-based named entity recognition (NER) technology is introduced. Specifically, the method includes:

[0041] First, perform data annotation: Manually or semi-manually structure the annotation of a part of ancient book texts using the BIO annotation rule. The BIO system is a rule commonly used in sequence annotation tasks, where 'B-SYM' represents the start of a symptom entity, 'I-SYM' represents the inside of a symptom entity, and 'O' represents the non-symptom part. For example, in "headache and aversion to cold", "head" is annotated as 'B-SYM', "ache" as 'I-SYM', "aversion" as 'B-SYM', and "cold" as 'I-SYM'. For single-word symptoms such as "cough", the 'S-SYM' label can be used. By performing such annotations on thousands of sentences (such as more than 3000 sentences) of ancient book texts, a high-quality training dataset is constructed.

[0042] In some preferred embodiments, the BIO annotation of ancient books is in JSON format, including two main fields:'sentence_index' (sentence number) and'result' (annotation result). Among them, 'result' contains 'originalText' (original text) and 'entities' (entity list), and each entity is annotated with'start_pos' (starting position), 'end_pos' (ending position), and 'label_type' (entity type, such as "symptom"). For example, "developing pox" in the annotation is marked as a symptom, and the position is from the 22nd to the 24th character. The annotation range is accurate to the character level, and only key entities are annotated. This method can annotate symptom entities.

[0043] Next, model training and extraction are performed: Using the aforementioned annotated dataset, a pre-trained language model for TCM (e.g., a BERT model pre-trained on a large corpus of domain data, such as TCM medical records and modern clinical guidelines) is fine-tuned. After sufficient training, this model deeply understands the linguistic paradigms of ancient TCM texts and can serve as a highly accurate symptom recognizer, automatically identifying and extracting initial symptom descriptions from large amounts of unannotated ancient texts to form an initial symptom list.

[0044] S2. For the complex symptoms in the initial symptom description, a preset rule engine and a large language model are used to split them to obtain a single symptom description.

[0045] Symptom descriptions directly extracted from ancient texts may sometimes be complex symptoms composed of multiple single symptoms. For example, "headache and neck pain" is actually a combination of "headache" and "neck pain." To achieve accurate standardization, these complex symptoms must first be decomposed. This paper proposes a two-layer decomposition mechanism that combines a rule engine with a large language model (LLM). Its advantage lies in balancing processing efficiency and breadth of coverage. Specifically, it includes:

[0046] S21. Construct the rule engine based on a preset body part dictionary and / or TCM grammatical rules to split the complex symptoms composed of clear body part words. The rule engine is used to process common complex symptoms with simple structures and obvious patterns. The rule can be based on a preset body part dictionary (such as "head", "neck", "chest", "abdomen", "hands", "feet", etc.) and / or common grammatical rules in the field of TCM. For example, set a rule as "when two or more body part nouns appear consecutively and are followed by expressions such as 'pain', 'swelling', and 'numbness', split them into their own independent symptoms." In this way, the rule engine can quickly and accurately split "head and neck pain" into "headache" and "neck pain".

[0047] S22. Build a prompt template and use the large language model to split the complex symptoms that the rule engine fails to handle. For long-tail complex symptoms that are not covered by the rule engine and have a more complex or rare structure, call a large language model (such as DeepSeek, GPT series, etc.) for processing. Specifically, when implementing, one or more dedicated prompt templates can be designed. Input the complex symptoms to be split into the large language model (such as by calling through an API interface), and require it to return a list of split single symptoms in a specified format. For example, the prompt template can be: "Please split the following traditional Chinese medicine symptom word '{symptom}' into the most basic and independent single symptoms and return them in a list form. If it is already a single symptom, directly return the original word." Inputting "chest and hypochondrium fullness, epigastric discomfort, loss of appetite", the large model can understand its semantics and may split it into multiple independent symptom expressions such as "chest and hypochondrium fullness", "epigastric discomfort", "loss of appetite".

[0048] Through this intelligent way of human-machine collaboration, it not only ensures the efficient processing of high-frequency common complex symptoms but also uses the powerful language understanding ability of the large model to solve difficult and rare splitting problems, thus obtaining a purer list of single symptom expressions.

[0049] S3. Calculate the multi-dimensional similarity of the single symptom expressions based on a preset standard symptom word library. The multi-dimensional similarity includes: structural similarity based on the edit distance; direct semantic similarity obtained based on the first pre-trained language model; and translated semantic similarity obtained by converting the single symptom expressions from ancient book terms to modern terms and using the second pre-trained language model.

[0050] Among them, as Figure 2 shown, the standard symptom word library can be based on national or industry authoritative standards, such as 《GB / T15657 Classification and Codes of Traditional Chinese Medicine Syndromes》, etc., and can integrate other knowledge bases (such as TCM-KD). The internal structure of the word library is built as a multi-level mapping system, including standard words (such as general normative terms in modern medicine or traditional Chinese medicine circles), formal names (more standardized expressions in ancient books), and aliases (synonyms, local dialects, variant characters, etc.), forming a structured knowledge network.

[0051] The structural similarity focuses on the literal structural differences of symptom expressions and uses a classic edit distance algorithm (such as the Levenshtein distance) to calculate the string differences between the symptoms to be normalized and the standard symptoms. The edit distance represents the minimum number of single-character edits (insertions, deletions, replacements) required to convert one string to another. This dimension is very effective for handling similar symptoms caused by copying errors or minor differences in the use of words (such as "the head feels heavy as if wrapped" and "the head feels heavy as though wrapped"). The calculation result constitutes the structural similarity score WModel1 。

[0052] Direct semantic similarity captures associations that go beyond literal meaning and delve into semantic depth. In some preferred embodiments, it can be obtained using the following methods:

[0053] S301. Fine-tune a language model using a dataset that includes pairs of standard symptom terms and ancient medical symptom terms to obtain the first pre-trained language model. The language model can be a BERT model. In this way, the model learns the deep semantic mapping relationships between ancient and modern, and synonymic symptoms. For example, "head disease" and "headache" are semantically equivalent.

[0054] S302. Use the first pre-trained language model to convert the single symptom expression and the standard symptom into vector representations, and calculate the cosine similarity between the two vectors to quantify their proximity in the semantic space. The closer the cosine value is to 1, the more similar the semantics. The calculation result constitutes the direct semantic similarity score W Model2 。

[0055] Translated semantic similarity is designed specifically to handle symptoms where there are significant differences between ancient medical terms and modern terms, making direct semantic matching difficult. Its specific implementation steps include:

[0056] S311. Fine-tune a neural machine translation (NMT) model (e.g., based on the Transformer architecture) using a dataset that includes traditional Chinese medicine symptoms and their corresponding modern expressions. Use the fine-tuned neural machine translation model to convert the single symptom expression from ancient medical terms to modern terms, thereby translating the ancient medical symptom expression into understandable modern terms. For example, "difficulty in the anal tract" may be translated as "constipation" or "inability to defecate smoothly".

[0057] S312. Fine-tune a language model (the language model here can be the same or of the same type as the model in step S301, but essentially it is another language model) using a triple-paired dataset that includes standard symptom terms, corresponding modern expressions, and pairs of ancient medical symptoms to obtain the second pre-trained language model.

[0058] S313. Use the second pre-trained language model to convert the single symptom expression after term conversion and the standard symptom into vector representations, and calculate the cosine similarity between the two vectors. This strategy of comparing after translation greatly improves the ability to understand and match complex and obscure ancient medical expressions. The calculation result constitutes the translated semantic similarity score W Model3 。

[0059] S4. Perform weighted fusion on the multi-dimensional similarities to obtain a comprehensive similarity score, and determine a corresponding standard symptom from the standard symptom vocabulary based on the comprehensive similarity score, and use it as a standardized expression of the initial symptom statement.

[0060] The fusion process can be expressed by a weighted formula:

[0061] ;

[0062] in, ,β,γ are their respective weight coefficients, and satisfy These weights can be set to initial values ​​based on experience. For example, since semantic information is usually more important than character structure, a higher weight can be given to the semantic model (e.g. ). Preferably, these weights can also be dynamically adjusted based on the performance evaluation results of the model on the validation set (such as the F1 value) to achieve the best normalization effect.

[0063] For each single symptom description, the overall similarity score between it and all candidate standard symptoms in the standard vocabulary is calculated, and the standard symptom with the highest score is determined as the final standardized expression. This completes the mapping from the initial symptom description in the ancient text to the modern standard terminology.

[0064] The basic principles, main features, and advantages of the present invention are shown and described above. Those skilled in the art should understand that the present invention is not limited to the foregoing embodiments. The foregoing embodiments and descriptions are merely illustrative of the principles of the present invention. Various changes and modifications may be made to the present invention without departing from the spirit and scope of the present invention. Such changes and modifications are intended to fall within the scope of the present invention. The scope of protection claimed in the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for standardizing symptoms in ancient Chinese medical books based on a large language model, characterized by: The steps include: S1. Obtain and preprocess ancient Chinese medical texts to extract initial symptom descriptions; S2. Split the complex symptoms in the initial symptom description using a combination of a preset rule engine and a large language model to obtain a single symptom description; S3. Calculating the multidimensional similarity of the single symptom expression based on a preset standard symptom vocabulary, the multidimensional similarity comprising: structural similarity based on edit distance; direct semantic similarity obtained using a first pre-trained language model; and post-translation semantic similarity obtained by converting the single symptom expression from ancient texts to modern terms using a second pre-trained language model; S4. performing weighted fusion on the multi-dimensional similarities to obtain a comprehensive similarity score, and determining a corresponding standard symptom from the standard symptom vocabulary based on the comprehensive similarity score, and using it as a standardized expression of the initial symptom description; The method for extracting the initial symptom expression in step S1 includes: using BIO annotation rules to perform structured annotation on the ancient Chinese medical text, and using a TCM pre-trained model trained with the annotated data to identify and extract the initial symptom expression; Step S2 includes: Building the rule engine based on a preset body part dictionary and / or TCM grammar rules to split compound symptoms composed of specific body part words; Construct prompt word templates and use the large language model to split complex symptoms that the rule engine cannot handle.

2. The method for standardizing symptoms in ancient Chinese medical books based on a large language model as claimed in claim 1, characterized in that: The method for calculating the direct semantic similarity includes: Fine-tuning the language model using a dataset containing pairings of standard symptom terms with symptoms from ancient books to obtain the first pre-trained language model; The single symptom expression and the standard symptom are converted into vector representations using the first pre-trained language model, and the cosine similarity between the two vectors is calculated.

3. The method for standardizing symptoms in ancient Chinese medical books based on a large language model as claimed in claim 1, characterized in that: The method for calculating the translated semantic similarity includes: Fine-tuning a neural machine translation model using a dataset containing TCM symptoms and their corresponding modern terms, and using the fine-tuned neural machine translation model to convert the single symptom description from ancient Chinese medicine terms to modern terms; Fine-tune the language model using a dataset containing standard symptom terms, corresponding modern terms, and symptom pairs from ancient books to obtain the second pre-trained language model; The second pre-trained language model is used to convert the single symptom expression after terminology conversion and the standard symptom into vector representations, and the cosine similarity between the two vectors is calculated.

4. The method for standardizing symptoms in ancient Chinese medical books based on a large language model as claimed in claim 1, characterized in that: The method for constructing the preset standard symptom vocabulary includes: constructing a multi-level mapping system including standard words, correct names and aliases based on national standards and / or industry standard terminology sets.

5. The method for standardizing symptoms in ancient Chinese medical books based on a large language model as claimed in claim 1, characterized in that: The step of pre-processing the ancient Chinese medical text in step S1 includes: removing non-text symbols in the ancient Chinese medical text using regular expressions, and unifying variant characters or homophones in the text based on a variant character comparison table.

Images