Traditional Chinese medicine ancient book word segmentation method and device, computer device and storage medium

Abstract

The application relates to a traditional Chinese medicine ancient book word segmentation method and device, computer equipment and a storage medium in the technical field of Chinese word segmentation, which comprises the following steps: constructing an ancient and modern word meaning correspondence table, a rule table, a self-defined term library and a traditional Chinese medicine field term dictionary; adopting a rule-based method to translate a to-be-segmented text from ancient Chinese into modern Chinese according to the ancient and modern word meaning correspondence table, the rule table, the self-defined term library and the traditional Chinese medicine field term dictionary; performing word segmentation processing on the translated text by adopting a traditional Chinese medicine field term dictionary assisted Jieba word segmentation tool; solving an unregistered word problem by using a hidden Markov model to obtain a word segmentation result; and translating the word segmentation result into an original text according to the ancient and modern word meaning correspondence table and the rule table to obtain a final word segmentation text. The method effectively improves the word segmentation accuracy of traditional Chinese medicine ancient books, alleviates the problem of poor word segmentation effect caused by ancient and modern language differences and field differences, and provides technical support for digitalization of traditional Chinese medicine.

Description

Technical Field

[0001] This application relates to the field of Chinese word segmentation technology, and in particular to a method, apparatus, computer equipment, and storage medium for segmenting words in ancient Chinese medicine texts. Background Technology

[0002] Informatization of Traditional Chinese Medicine (TCM) is a crucial aspect of current TCM research, and information technology is a powerful force driving its development. In the field of TCM, Chinese word segmentation, as the foundation of domain-specific text information processing, has two main significances: first, it lays the groundwork for tasks such as text translation and intelligent question answering; second, it provides support for understanding and disseminating TCM texts. However, due to the unique characteristics of the language and structure of ancient TCM texts, the overall word segmentation results are relatively poor.

[0003] Chinese word segmentation refers to dividing a sentence into multiple words, that is, dividing a long sequence into multiple consecutive semantically independent short sequences. The segmentation rules are composed of factors such as part of speech and word usage habits. In the field of traditional Chinese medicine, there is relatively little research on text segmentation. For example, Zhang Fan et al. studied a word segmentation scheme suitable for TCM medical records and found that introducing a TCM dictionary greatly helped with the segmentation of medical records, but the automatic word segmentation performance of ancient medical records was lower than that of modern medical records. Yang Haifeng et al. explored word segmentation tools suitable for TCM literature and gave a better choice for TCM literature segmentation. Wang Lijun et al. studied word segmentation models suitable for TCM classical texts and found that the BiLSTM-CRF model had better segmentation performance.

[0004] Traditional Chinese medicine (TCM) classics are written in Classical Chinese, and their textual structure and semantics differ significantly from modern Chinese. Furthermore, the texts contain numerous function words, abbreviations, inverted words, and technical terms, increasing the difficulty of word segmentation. Additionally, linguistic differences across dynasties and regions make word segmentation training for TCM classics challenging and costly. Chinese word segmentation is a crucial foundation for Chinese information processing. Due to the unique language and writing rules of TCM classics, existing mature word segmentation tools cannot meet the needs of information processing research on these texts. Summary of the Invention

[0005] Therefore, it is necessary to provide a method, apparatus, computer equipment, and storage medium for word segmentation of ancient Chinese medicine books to address the aforementioned technical problems.

[0006] A method for word segmentation of ancient Chinese medicine texts, the method comprising:

[0007] Obtain the text to be segmented from ancient Chinese medicine books.

[0008] Construct a table comparing the meanings of words in ancient and modern times, a rule table, a custom terminology database, and a terminology dictionary in the field of traditional Chinese medicine.

[0009] Based on the ancient and modern word meaning comparison table, the rule table, the custom terminology database, and the dictionary of terms in the field of traditional Chinese medicine, the text to be segmented is translated from ancient Chinese to modern Chinese using a rule-based method to obtain the translated text. The translated text includes: translation information of the translated word, original position information of the translated word, and the serial number of the translated word in the ancient and modern word meaning comparison table.

[0010] The translated text was segmented using the Jieba word segmentation tool, which is assisted by the terminology dictionary in the field of traditional Chinese medicine. Then, a hidden Markov model was used to solve the out-of-vocabulary word problem, and the word segmentation results were obtained.

[0011] The word segmentation results are translated into the original text based on the ancient and modern word meaning comparison table and the rule table to obtain the final segmented text.

[0012] In one embodiment, a glossary of ancient and modern word meanings, a rule table, a custom terminology database, and a dictionary of terms in the field of traditional Chinese medicine are constructed, including:

[0013] This project collects terms related to Chinese medicine, prescriptions, syndromes, diseases, symptoms, treatments, drug processing methods, and the human body from ancient Chinese medicine books, and constructs a terminology dictionary for the field of Chinese medicine based on the collected terms. The ancient Chinese medicine books mentioned include: "Handbook of Commonly Used Chinese Medicine Names and Alternative Names", "Standardized Terminology for Common Symptoms in Chinese Medicine Clinical Practice", "Comprehensive Dictionary of Chinese Medicine", and "Terminology for Clinical Diagnosis and Treatment in Chinese Medicine".

[0014] Based on the high-frequency translated words and annotations in existing translations of ancient Chinese medicine books, as well as ancient Chinese words and their translations and classical Chinese function words and their translations found online, a glossary of ancient and modern word meanings is constructed. The glossary includes terms from a dictionary of terms in the field of traditional Chinese medicine and fixed vocabulary in the field of traditional Chinese medicine.

[0015] Classical Chinese words requiring polysemous word modification are included in the rule table. The polysemous word translation rule is as follows: if a keyword appears in the preceding / following character of the classical Chinese word to be translated, then the translation is performed according to the modern Chinese word in the corresponding row of the keyword; if no keyword appears in the preceding / following character of the classical Chinese word, then the translation is performed according to the definition of the classical Chinese word in the ancient and modern word meaning comparison table; the keyword is derived from the original corpus.

[0016] A custom TCM terminology database was constructed based on the aforementioned TCM terminology dictionary, book titles, personal names, dynasty names, and commonly used words. The book titles, personal names, and dynasty names were collected from the internet. The commonly used words were obtained by crawling Baidu Dictionary using BeautifulSoup, and the crawled content consisted of related words of Chinese characters that required partial modification.

[0017] In one embodiment, based on the ancient-modern word meaning comparison table, the rule table, the custom terminology database, and the dictionary of terms in the field of traditional Chinese medicine, a rule-based method is used to translate the text to be segmented from classical Chinese to modern Chinese, including:

[0018] The terminology dictionary in the field of traditional Chinese medicine is sorted from longest to shortest based on the length of the term strings, resulting in a sorted dictionary of terms in the field of traditional Chinese medicine.

[0019] The words to be translated and their corresponding position information are extracted from the ancient and modern word meaning comparison table and the sorted dictionary of terms in the field of traditional Chinese medicine.

[0020] The extracted words that need to be translated are translated according to the ancient and modern word meaning comparison table, the rule table, and the custom terminology database to obtain the translated text; the translated text includes: the translation information of the translated word, the original position information of the translated word, and the serial number of the translated word in the ancient and modern word meaning comparison table.

[0021] In one embodiment, the extracted words to be translated are translated according to the ancient and modern word meaning comparison table, the rule table, and the custom terminology database to obtain the translated text, including:

[0022] Based on the ancient and modern word meaning comparison table, a preliminary selection of words that need to be translated in classical Chinese is made.

[0023] The system determines whether the words to be translated need to be translated based on the custom terminology database, and obtains the result.

[0024] If the judgment result is that translation is not required, then no translation will be performed.

[0025] If the determination result indicates that translation is required, and the extracted words to be translated are words that need to be modified, if it is a polysemous word modification, then the extracted words to be translated are translated according to the custom terminology rule table and the ancient and modern word meaning comparison table to obtain the translated text; if it is a partial modification, then words not in the custom input library are modified to obtain the translated text.

[0026] In one embodiment, the word segmentation result is translated into the original text according to the ancient and modern word meaning comparison table and the rule table to obtain the final segmented text, including:

[0027] When the translated word in the word segmentation result is the original word, the final segmented text is obtained;

[0028] When the translated word in the word segmentation result is not the original word:

[0029] If the translated words are segmented, the delimiters are removed, and then the translated words in the segmentation results are translated according to the ancient and modern word meaning comparison table and the rule table to obtain the final segmented text;

[0030] If the translated words are not segmented, the translated words in the segmentation results are translated according to the ancient and modern word meaning comparison table and the rule table to obtain the final segmented text.

[0031] A word segmentation device for ancient Chinese medicine texts, the device comprising:

[0032] The data acquisition module is used to acquire the text to be segmented from ancient Chinese medicine books.

[0033] The translation data table and rule building module is used to construct ancient and modern word meaning comparison tables, rule tables, custom terminology databases, and a terminology dictionary in the field of traditional Chinese medicine.

[0034] The local translation module is used to translate the text to be segmented from classical Chinese to modern Chinese using a rule-based method, based on the ancient and modern word meaning comparison table, the rule table, the custom terminology database, and the dictionary of terms in the field of traditional Chinese medicine, to obtain the translated text. The translated text includes: translation information of the translated word, original position information of the translated word, and the serial number of the translated word in the ancient and modern word meaning comparison table.

[0035] The word segmentation module is used to segment the translated text with the aid of the Jieba word segmentation tool, which is based on the terminology dictionary in the field of traditional Chinese medicine. Then, it uses a hidden Markov model to solve the out-of-vocabulary word problem and obtain the word segmentation result.

[0036] The restoration module is used to translate the word segmentation results into the original text according to the ancient and modern word meaning comparison table and the rule table, so as to obtain the final word segmentation text.

[0037] In one embodiment, the translation data table and rule-building module is also used to collect terms related to Chinese medicine, prescriptions, syndromes, diseases, symptoms, treatments, drug processing methods, and the human body from ancient Chinese medicine books, and to construct a terminology dictionary for the field of Chinese medicine based on the collected terms; the ancient Chinese medicine books include: "Handbook of Commonly Used Chinese Medicine Names and Alternative Names", "Standardized Terminology for Common Symptoms in Chinese Medicine Clinical Practice", "Dictionary of Chinese Medicine", and "Terminology for Clinical Diagnosis and Treatment in Chinese Medicine"; based on high-frequency translated words and annotations in existing translations of ancient Chinese medicine books, as well as ancient Chinese words and their translations and classical Chinese function words and their translations found online, a glossary of ancient and modern word meanings is constructed; the glossary of ancient and modern word meanings includes: terms in the terminology dictionary for the field of Chinese medicine and fixed vocabulary in the field of Chinese medicine; and will include more... The classical Chinese words requiring semantic modification are included in the rule table. The translation rules for polysemous words are as follows: if a keyword appears in the preceding / following character of the classical Chinese word to be translated, it is translated according to the modern Chinese word in the corresponding row; if no keyword appears in the preceding / following character of the classical Chinese word, it is translated according to the definition of the classical Chinese word in the ancient-modern word meaning comparison table; the keywords are derived from the original corpus; a custom terminology database for traditional Chinese medicine is constructed based on the terminology dictionary, book titles, personal names, dynasty names, and commonly used words in the field of traditional Chinese medicine; the book titles, personal names, and dynasty names are collected from the Internet; the commonly used words are obtained by crawling Baidu Dictionary using BeautifulSoup, and the crawled content is the related words of the Chinese characters that need to be partially modified.

[0038] In one embodiment, the local translation module is further configured to sort the terminology strings in the TCM terminology dictionary from longest to shortest to obtain a sorted TCM terminology dictionary; extract the words to be translated and their corresponding position information from the text to be segmented based on the ancient-modern word meaning comparison table and the sorted TCM terminology dictionary; translate the extracted words to be translated based on the ancient-modern word meaning comparison table, the rule table, and the custom terminology database to obtain the translated text; the translated text includes: translation information of the translated word, original position information of the translated word, and the serial number of the translated word in the ancient-modern word meaning comparison table.

[0039] A computer device includes a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the steps of any of the methods described above.

[0040] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the methods described above.

[0041] The aforementioned method, apparatus, computer equipment, and storage medium for word segmentation of ancient Chinese medicine texts include the following steps: acquiring the text to be segmented from the ancient Chinese medicine texts; constructing a glossary of ancient and modern Chinese word meanings, a rule table, a custom terminology database, and a dictionary of terms in the field of Chinese medicine; translating the text from classical Chinese to modern Chinese using a rule-based method based on the glossary of ancient and modern Chinese word meanings, the rule table, the custom terminology database, and the dictionary of terms in the field of Chinese medicine, obtaining the translated text; using the Jieba word segmentation tool assisted by the dictionary of terms in the field of Chinese medicine to perform word segmentation on the translated text, and then using a Hidden Markov Model to solve the out-of-vocabulary word problem, obtaining the word segmentation result; and translating the word segmentation result back into the original text based on the glossary of ancient and modern Chinese word meanings and the rule table, obtaining the final segmented text. This method effectively improves the word segmentation accuracy of ancient Chinese medicine texts, alleviates the problem of poor word segmentation results caused by differences in ancient and modern languages ​​and field differences, and provides technical support for the digital construction of traditional Chinese medicine. Attached Figure Description

[0042] Figure 1 This is a flowchart illustrating a word segmentation method for ancient Chinese medicine texts in one embodiment;

[0043] Figure 2 This is a technical approach to a word segmentation method for ancient Chinese medicine texts in one embodiment;

[0044] Figure 3 This is a schematic diagram of the local translation process in one embodiment;

[0045] Figure 4 This is a structural block diagram of a word segmentation device for ancient Chinese medicine texts in one embodiment;

[0046] Figure 5 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0047] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0048] This method first translates the classical Chinese texts in the ancient Chinese medicine books to be segmented into modern Chinese, and then introduces a segmentation strategy that uses a dictionary of terms in the field of Chinese medicine to assist in segmentation, so as to reduce the occurrence of segmentation errors caused by differences in the writing style and vocabulary between classical Chinese and modern Chinese.

[0049] In one embodiment, such as Figure 1 , Figure 2 As shown, a word segmentation method for ancient Chinese medicine texts is provided. The application of this method includes the following steps:

[0050] Step 100: Obtain the text to be segmented from ancient Chinese medicine books.

[0051] Specifically, the text to be segmented is from ancient Chinese medicine books, written in classical Chinese. These ancient Chinese medicine books are written in classical Chinese, and their structure, semantics, and other aspects differ significantly from modern Chinese. Furthermore, the texts contain a large number of function words, abbreviations, inverted words, and technical terms, which increases the difficulty of segmenting them.

[0052] Step 102: Construct a table comparing the meanings of words in ancient and modern times, a rule table, a custom terminology database, and a dictionary of terms in the field of traditional Chinese medicine.

[0053] Specifically, the glossary of ancient and modern word meanings and the rule table are crucial for text translation and text reinterpretation. The glossary of ancient and modern word meanings primarily comes from two sources: first, the internet, including common classical Chinese words and their translations, and classical Chinese function words and their translations; second, existing translations of traditional Chinese medicine texts, including high-frequency words from ancient books such as the *Huangdi Neijing* and *Shanghan Lun*. The rule table includes classical Chinese words requiring "polysemous word modification," as well as translation rules for polysemous words. Each record in the glossary of ancient and modern word meanings includes: serial number, classical Chinese, modern Chinese, and modification method; the modification methods are: 0 indicates modification only of the beginning of the sentence, 1 indicates modification of all, 2 indicates modification of polysemous words, 3 indicates partial modification, and 4 indicates modification only of the end of the sentence. Partial modification indicates modification only of words not in the terminology database; for classical Chinese words requiring polysemous word modification, only the most commonly used translation standards are listed, and other translation schemes refer to the rule table.

[0054] The custom terminology database consists of a dictionary of terms in the field of traditional Chinese medicine, book titles, personal names, dynasty names, and commonly used words.

[0055] The vocabulary in the dictionary of terms in the field of traditional Chinese medicine mainly comes from the "Handbook of Commonly Used Chinese Medicine Names and Synonyms", "Standardized Terminology for Common Symptoms in Traditional Chinese Medicine Clinical Practice", "Dictionary of Traditional Chinese Medicine", and "Terminology for Clinical Diagnosis and Treatment in Traditional Chinese Medicine".

[0056] Step 104: Based on the ancient and modern word meaning comparison table, rule table, custom terminology database, and dictionary of terms in the field of traditional Chinese medicine, the text to be segmented is translated from ancient Chinese to modern Chinese using a rule-based method to obtain the translated text. The translated text includes: translation information of the translated word, original position information of the translated word, and the serial number of the translated word in the ancient and modern word meaning comparison table.

[0057] Specifically, the translation from ancient Chinese to modern Chinese adopts a rule-based method for literal translation, with reference to the basic translation strategies of retention, substitution, adjustment, supplementation, deletion, and coherence, and applies the translation methods of retention, substitution and deletion, that is, retaining proprietary terms, substituting monosyllabic words / polysemous words, and deleting meaningless words. The specific practices are as follows: first, extract the vocabularies to be translated and their position information with the help of the ancient-modern Chinese meaning comparison table and the traditional Chinese medicine domain term dictionary, and then translate the extracted words according to the ancient-modern Chinese meaning comparison table, the rule table and the custom term base. Considering the needs of subsequent translation, the translated sentences also include the original position information of the translated words and the serial numbers of the translated words in the ancient-modern Chinese meaning comparison table.

[0058] Step 106: Use the traditional Chinese medicine domain term dictionary to assist the Jieba segmentation tool to perform word segmentation processing on the translated text, and then use the Hidden Markov Model to solve the out-of-vocabulary word problem to obtain a word segmentation result.

[0059] Specifically, the Jieba 0.42 is selected as the word segmentation tool, because the implementation mode of Jieba word segmentation is to first perform text word segmentation according to the custom lexicon, and then use the Hidden Markov Model to solve the out-of-vocabulary (OOV) problem, which can better retain the relevant term vocabularies. The present invention also introduces a custom term base during word segmentation to further improve the accuracy of word segmentation.

[0060] Step 108: Translate the word segmentation result into the original text according to the ancient-modern Chinese meaning comparison table and the rule table, so as to obtain the final segmented text.

[0061] Specifically, the word segmentation result is represented by the translated text, and the translated text finally needs to be translated back into the original text, and the implementation of the translation relies on the ancient-modern Chinese meaning comparison table. During the translation, if the translated text is segmented, the original text is retained, for example, "méiyǒu / dìfāng" (no place) is restored to " / wúsuǒ / " (nowhere).

[0062] The word segmentation method of performing partial translation first and then introducing a dictionary to assist word segmentation alleviates the problem of poor word segmentation effect caused by the differences between ancient and modern languages and the differences in technical fields to a certain extent.

[0063] The aforementioned method for word segmentation of ancient Chinese medicine texts includes: acquiring the text to be segmented from these texts; constructing a glossary of ancient and modern meanings, a rule table, a custom terminology database, and a dictionary of terms related to traditional Chinese medicine; translating the text from classical Chinese to modern Chinese using a rule-based approach, based on the glossary, rule table, custom terminology database, and dictionary of terms related to traditional Chinese medicine, to obtain the translated text; using the dictionary of terms related to traditional Chinese medicine to assist the Jieba word segmentation tool in segmenting the translated text, and then using a Hidden Markov Model to solve the out-of-vocabulary (OOV) word problem, to obtain the segmentation result; and finally, translating the segmentation result back into the original text based on the glossary and rule table, to obtain the final segmented text. This method effectively improves the accuracy of word segmentation for ancient Chinese medicine texts, alleviates the problem of poor segmentation results caused by differences in ancient and modern language and domain differences, and provides technical support for the digitalization of traditional Chinese medicine.

[0064] In one embodiment, step 102 includes: collecting terms related to Chinese medicine, prescriptions, syndromes, diseases, symptoms, treatments, drug processing methods, and the human body from ancient Chinese medicine books; and constructing a terminology dictionary for the field of Chinese medicine based on the collected terms; the ancient Chinese medicine books include: "Handbook of Commonly Used Chinese Medicine Names and Alternative Names," "Standardized Terminology for Common Symptoms in Traditional Chinese Medicine Clinical Practice," "Comprehensive Dictionary of Traditional Chinese Medicine," and "Terminology for Clinical Diagnosis and Treatment in Traditional Chinese Medicine"; constructing a glossary of ancient and modern meanings based on high-frequency translations and annotations in existing translations of ancient Chinese medicine books, as well as ancient Chinese words and their translations and classical Chinese function words and their translations found online; the glossary of ancient and modern meanings includes: terms from the terminology dictionary for the field of Chinese medicine and fixed vocabulary in the field of Chinese medicine; and compiling more... The classical Chinese words requiring semantic modification are included in the rule table. The translation rules for polysemous words are as follows: if the preceding / following characters of the classical Chinese word to be translated contain a keyword, then the translation is performed according to the modern Chinese of the corresponding row; if the preceding / following characters of the classical Chinese word do not contain a keyword, then the translation is performed according to the definition of the classical Chinese word in the ancient-modern word meaning comparison table; the keywords are derived from the original corpus; a custom terminology database for traditional Chinese medicine is constructed based on the terminology dictionary, book titles, personal names, dynasty names, and commonly used words in the field of traditional Chinese medicine; the book titles, personal names, and dynasty names are collected from the Internet; the commonly used words are obtained by crawling Baidu Dictionary using BeautifulSoup, and the crawled content consists of related words of the Chinese characters that need to be partially modified.

[0065] Specifically, the number of words included in each category of the dictionary of terms in the field of traditional Chinese medicine is shown in Table 1.

[0066] Table 1. Dictionary of Terms in the Field of Traditional Chinese Medicine

[0067] traditional Chinese medicine 2429 symptom 2461 prescription 917 treatment 1805 Certificate type 3199 Methods of drug processing and administration 53 disease 2035 human body 466

[0068] The ancient-modern meaning comparison table and the rule table are the key to text translation and text transposition. The vocabulary sources of the ancient-modern meaning comparison table mainly include two aspects: first, the Internet, including common classical Chinese words and their translations, classical Chinese function words and their translations; second, existing translations of traditional Chinese medicine, including high-frequency translated words and annotated words in ancient books such as Huangdi Neijing and Shanghan Lun. In addition, to ensure the fluency of the translation and the needs of subsequent word segmentation, in addition to retaining 8 types of terms in the term dictionary, fixed terms in the field of traditional Chinese medicine such as "vital qi" and "nutritive and defensive qi" are not included in the ancient-modern meaning comparison table. Finally, a total of 145 ancient-modern comparison groups were collected and sorted out, and the collected examples are shown in Table 2. The column "How to modify" in the table: 0 means only modify the beginning of the sentence, 1 means modify the whole, 2 means modify polysemous words, 3 means partial modification, 4 means only modify the end of the sentence. Among them, "partial modification" means only modifying the words not in the term base. For the classical Chinese that needs "polysemous word modification", only the most commonly used translation specification is listed, and the rest of the translation schemes refer to the rule table.

[0069] Table 2 Ancient-modern Meaning Comparison Table

[0070] 1 Ear - 4 2 square prescription 0 3 say explain 1 4 Of of 2 5 white Always 3

[0071] Note: "-" means empty

[0072] The rule table collects classical Chinese words that require "polysemous word modification". The translation rules for polysemous words are as follows: if a "keyword" appears in the preceding / following character of the classical Chinese to be translated, the translation is carried out according to the modern Chinese corresponding to the row of the keyword; if no "keyword" appears in the preceding / following character of the classical Chinese, the translation is carried out according to the definition of the classical Chinese in the "ancient-modern meaning comparison table". Among them, the "keywords" are derived from the original corpus, and the specific method is as follows: the machine first automatically extracts one character before and after the classical Chinese that needs polysemous word modification, integrates and eliminates low-frequency words (low-frequency words refer to words with a frequency less than or equal to 3), then manually classifies the keywords, and the example of the result is shown in Table 3. The foreign key in the table represents the position of the classical Chinese in the ancient-modern meaning comparison table.

[0073] Table 3 Rule File Table

[0074] 1 4 Of of Job, function, medicine, prescription, sweet, product 2 4 Of - Deficiency, interior, method, qi, pulse, syndrome 3 4 Of it Vomiting, treating, expelling, eliminating, taking, main

[0075] Note: Taking "zhi" as an example, "-" means empty

[0076] The custom term base is composed of term dictionaries in the field of traditional Chinese medicine, book names, person names, dynasty names, common words, etc., and contains a total of 14969 words (single characters are excluded). Among them, book names, person names, dynasty names, etc. are collected from the Internet; common words are obtained by crawling Baidu Vocabulary with BeautifulSoup, and the crawled content is the related word groups of the characters that "require partial modification" in Table 2.

[0077] In one embodiment, step 104 includes: sorting the terminology strings in the TCM terminology dictionary from longest to shortest length to obtain a sorted TCM terminology dictionary; extracting the words to be translated and their corresponding position information from the text to be segmented based on the ancient-modern word meaning comparison table and the sorted TCM terminology dictionary; translating the extracted words to be translated based on the ancient-modern word meaning comparison table, the rule table, and the custom terminology database to obtain the translated text; the translated text includes: translation information of the translated word, the original position information of the translated word, and the serial number of the translated word in the ancient-modern word meaning comparison table. The partial translation process is as follows: Figure 3 As shown.

[0078] In one embodiment, the steps are as follows: Translate the extracted words that need to be translated according to the ancient and modern word meaning comparison table, the rule table, and the custom terminology database to obtain the translated text. This includes: initially selecting words that need to be translated based on the ancient Chinese in the ancient and modern word meaning comparison table; determining whether the words that need to be translated need to be translated based on the custom terminology database, and obtaining a determination result; if the determination result is that they do not need to be translated, then no translation is performed; if the determination result is that they need to be translated, and the extracted words that need to be translated are words that need to be modified, if it is a polysemous word modification, then translate the extracted words that need to be translated according to the custom terminology rule table and the ancient and modern word meaning comparison table to obtain the translated text; if it is a partial modification, then modify the words that are not in the custom input database to obtain the translated text.

[0079] In one embodiment, step 108 includes: when the translated word in the segmentation result is the original word, the final segmented text is obtained; when the translated word in the segmentation result is not the original word: if the translated word is segmented, the delimiter is removed, and the translated word in the segmentation result is translated according to the ancient and modern word meaning comparison table and the rule table to obtain the final segmented text; if the translated word is not segmented, the translated word in the segmentation result is translated according to the ancient and modern word meaning comparison table and the rule table to obtain the final segmented text.

[0080] It should be understood that, although Figure 1 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figure 1 At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0081] In a verification example, a new word segmentation scheme suitable for the current status of research on word segmentation of ancient Chinese medicine books is designed and implemented based on Jieba word segmentation, to verify the feasibility of the word segmentation scheme and the effectiveness of partial text translation in word segmentation of ancient Chinese medicine books.

[0082] Firstly, the word segmentation annotated corpus of "Shan Bu Yi Ming Fang Lun" (Supplemented and Revised Commentaries on Famous Physicians' Prescriptions) is constructed. The construction of the corpus refers to the People's Daily Annotated Corpus and the Word Segmentation Corpus of Qing Dynasty Medical Books. The word segmentation is manually performed by multiple professionals of traditional Chinese medicine, and the words comply with the standards in the field of Chinese medicine such as "Terminology of Traditional Chinese Medicine Clinical Diagnosis and Treatment", common symptom standard terms comply with "Terminology of Traditional Chinese Medicine Clinical Diagnosis and Treatment", common symptom standard terms comply with "Terminology of Traditional Chinese Medicine Clinical Diagnosis and Treatment", and common symptom norms. The word segmentation is carried out reasonably according to part of speech, meaning and vocabulary structure, and the word segmentation results are cross-corrected by experts to ensure the integrity of commonly used traditional Chinese medicine terms such as diseases, prescriptions, and syndromes in the word segmentation corpus and the accuracy of the annotation results. Words are separated by " / ", and the word segmentation annotation sample is as follows:

/ 集注 /

[0083] Taking the word segmentation dataset of "Shan Bu Yi Ming Fang Lun" as the test dataset, four experiments are set up, which are the original Jieba word segmentation, the Jieba word segmentation with an introduced dictionary, the Jieba word segmentation with introduced partial translation, and the Jieba word segmentation with both the introduced dictionary and introduced translation.

[0084] Table 4 lists the overall word segmentation effect and term recall of Jieba before and after the optimization of the word segmentation route. Before the optimization of the word segmentation technical route, the word segmentation performance of Jieba is not satisfactory, with an F1 value of only 51.55%, and problematic segmentations such as "以胃 / 气为 / 本 / " (taking stomach qi as the root) and "治实 / 热 / 老 / 痰 / 之峻剂 / " (a potent agent for treating excess heat and old phlegm) frequently occur; after optimization using the word segmentation scheme proposed in this study, "以胃气为本" is translated as "taking stomach qi as the root", and "治实热老痰之峻剂" is translated as "a potent agent for treating excess heat and old phlegm". In the case where only "excess heat" is included in the dictionary, the text can be accurately segmented into "以 / 胃气 / 为 / 本" and "治 / 实热 / 老痰 / 之 / 峻剂".

[0085] Meanwhile, it can be seen from Table 4 and the word segmentation results that compared with the method of only adding a dictionary, the word segmentation scheme that only introduces partial translation solves the problem of wrong segmentation caused by vocabulary differences between ancient and modern times to a greater extent, and plays a greater role in improving the F1 value; while adding a dictionary is more helpful to ensure the integrity of traditional Chinese medicine terms; therefore, this study combines the methods of partial translation and dictionary to obtain better word segmentation effects by integrating the advantages of the two methods.

[0086] The reasons for the low accuracy of word segmentation in ancient Chinese medicine texts are as follows: First, ancient Chinese medicine texts contain a large number of function words. In addition, the language of ancient Chinese medicine texts is concise and abstract, and the vocabulary of the texts differs greatly from that of modern Chinese, which easily leads to ambiguous segmentation such as " / Danpi / poxueyi / zhuqi / yu / " and "yi / shi zhen / xie". Second, the texts of Chinese medicine have prominent features such as inversion and abbreviation. For example, "dantou hanchu" should be "dantou chu han", and "mahuang, guizhi" is abbreviated to "ma gui", which increases the difficulty of text segmentation. Third, ancient Chinese medicine texts contain a large number of Chinese medicine terms, such as "xinxia pi ying", "fuzi xie xin tang", and "wei fa han", which have prominent domain characteristics and are difficult to segment accurately using existing word segmentation tools directly.

[0087] Table 4 Comparison of word segmentation results before and after word segmentation route optimization

[0088] P(％) 57.14 60.95 70.91 74.22 R(％) 46.96 49.16 67.06 68.87 F1(％) 51.55 54.43 68.93 71.45 R (%) 70.71 81.92 73.99 84.49

[0089] In one embodiment, such as Figure 4 As shown, a word segmentation device for ancient Chinese medicine texts is provided, comprising: a data acquisition module, a data table and rule construction module for translation, a local translation module, a word segmentation module, and a restoration module, wherein:

[0090] The data acquisition module is used to acquire the text to be segmented from ancient Chinese medicine books.

[0091] The translation data table and rule building module is used to construct ancient and modern word meaning comparison tables, rule tables, custom terminology databases, and a terminology dictionary in the field of traditional Chinese medicine.

[0092] The local translation module is used to translate the text to be segmented from classical Chinese to modern Chinese using a rule-based method, based on a table of ancient and modern word meanings, a rule table, a custom terminology database, and a dictionary of terms in the field of traditional Chinese medicine. The translated text includes: translation information of the translated word, original position information of the translated word, and the serial number of the translated word in the table of ancient and modern word meanings.

[0093] The word segmentation module uses a dictionary of terms in the field of traditional Chinese medicine to assist the Jieba word segmentation tool in segmenting the translated text, and then uses a hidden Markov model to solve the out-of-vocabulary word problem to obtain the word segmentation results.

[0094] The restoration module is used to translate the word segmentation results back into the original text based on the ancient and modern word meaning comparison table and the rule table, so as to obtain the final word segmentation text.

[0095] In one embodiment, the translation data table and rule building module are also used to collect terms related to Chinese medicine, prescriptions, syndromes, diseases, symptoms, treatments, drug processing methods, and the human body from ancient Chinese medicine books, and to build a terminology dictionary for the field of Chinese medicine based on the collected terms. The ancient Chinese medicine books include: *Handbook of Commonly Used Chinese Medicine Names and Alternative Names*, *Standardized Terminology for Common Symptoms in Traditional Chinese Medicine Clinical Practice*, *Comprehensive Dictionary of Traditional Chinese Medicine*, and *Terminology for Clinical Diagnosis and Treatment in Traditional Chinese Medicine*. Based on high-frequency translated words and annotations in existing translations of ancient Chinese medicine books, as well as ancient Chinese words and their translations and classical Chinese function words and their translations found online, a glossary of ancient and modern meanings is constructed. This glossary includes terms from the terminology dictionary for the field of Chinese medicine and fixed vocabulary in the field of Chinese medicine. The classical Chinese words requiring polysemous word modification are included in the rule table. The translation rules for polysemous words are as follows: if the keyword appears in the preceding / following character of the classical Chinese word to be translated, it is translated according to the modern Chinese of the corresponding row; if the keyword does not appear in the preceding / following character of the classical Chinese word, it is translated according to the definition of the classical Chinese word in the ancient and modern word meaning comparison table; the keywords are derived from the original corpus; a custom terminology database for traditional Chinese medicine is constructed based on the terminology dictionary, book titles, personal names, dynasty names, and commonly used words in the field of traditional Chinese medicine; the book titles, personal names, and dynasty names are collected from the Internet; the commonly used words are obtained by crawling Baidu Dictionary using BeautifulSoup, and the crawled content is the related words of the Chinese characters that need to be partially modified.

[0096] In one embodiment, the local translation module is further configured to sort the terminology strings in the TCM terminology dictionary from longest to shortest to obtain a sorted TCM terminology dictionary; extract the words to be translated and their corresponding position information from the text to be segmented in the ancient-modern word meaning comparison table and the sorted TCM terminology dictionary; translate the extracted words to be translated according to the ancient-modern word meaning comparison table, the rule table, and the custom terminology database to obtain the translated text; the translated text includes: translation information of the translated word, original position information of the translated word, and the serial number of the translated word in the ancient-modern word meaning comparison table.

[0097] In one embodiment, the partial translation module is further configured to initially select words to be translated based on the ancient Chinese semantic comparison table; determine whether the words to be translated need to be translated based on a custom terminology database, and obtain a judgment result; if the judgment result is that no translation is required, then no translation is performed; if the judgment result is that translation is required, and the extracted words to be translated are words that need to be modified, if it is a polysemous word modification, then the extracted words to be translated are translated according to the custom terminology rule table and the ancient Chinese semantic comparison table, and the translated text is obtained; if it is a partial modification, then words not in the custom input database are modified, and the translated text is obtained.

[0098] In one embodiment, the step restoration module is used to obtain the final segmented text when the translated word in the segmentation result is the original word; when the translated word in the segmentation result is not the original word: if the translated word is segmented, the delimiter is removed, and the translated word in the segmentation result is translated according to the ancient and modern word meaning comparison table and the rule table to obtain the final segmented text; if the translated word is not segmented, the translated word in the segmentation result is translated according to the ancient and modern word meaning comparison table and the rule table to obtain the final segmented text.

[0099] Specific limitations regarding the word segmentation device for ancient Chinese medicine texts can be found in the limitations of the word segmentation method for ancient Chinese medicine texts mentioned above, and will not be repeated here. Each module in the aforementioned word segmentation device for ancient Chinese medicine texts can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the corresponding operations of each module.

[0100] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 5 As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with external terminals via a network connection. When executed by the processor, the computer program implements a method for word segmentation of ancient Chinese medicine texts. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the computer device casing, or an external keyboard, touchpad, or mouse.

[0101] Those skilled in the art will understand that Figure 5 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0102] In one embodiment, a computer device is provided, including a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the steps in the above method embodiments.

[0103] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps in the above method embodiments.

[0104] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0105] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0106] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A method for word segmentation in ancient Chinese medicine texts, characterized in that, The method includes: Obtain the text to be segmented from ancient Chinese medicine books; Construct a table comparing ancient and modern word meanings, a rule table, a custom terminology database, and a terminology dictionary for the field of traditional Chinese medicine; Based on the ancient and modern word meaning comparison table, the rule table, the custom terminology database, and the dictionary of terms in the field of traditional Chinese medicine, the text to be segmented is translated from ancient Chinese to modern Chinese using a rule-based method to obtain the translated text. The translated text includes: translation information of the translated word, original position information of the translated word, and the serial number of the translated word in the ancient and modern word meaning comparison table. The translated text was segmented using the Jieba word segmentation tool assisted by the terminology dictionary in the field of traditional Chinese medicine. Then, a hidden Markov model was used to solve the out-of-vocabulary word problem to obtain the word segmentation results. The word segmentation results are translated into the original text based on the ancient and modern word meaning comparison table and the rule table to obtain the final word segmentation text; wherein, when the translated word in the word segmentation result is the original word, the final word segmentation text is obtained. When the translated word in the word segmentation result is not the original word: If the translated words are segmented, the delimiters are removed, and then the translated words in the segmentation results are translated according to the ancient and modern word meaning comparison table and the rule table to obtain the final segmented text; If the translated words are not segmented, the translated words in the segmentation results are translated according to the ancient and modern word meaning comparison table and the rule table to obtain the final segmented text.

2. The method according to claim 1, characterized in that, Construct a glossary of ancient and modern word meanings, a rule table, a custom terminology database, and a dictionary of terms in the field of traditional Chinese medicine, including: This project collects terminology related to Chinese medicine, prescriptions, syndromes, diseases, symptoms, treatments, drug processing methods, and the human body from ancient Chinese medicine books, and constructs a terminology dictionary for the field of Chinese medicine based on the collected terminology. The ancient Chinese medicine books mentioned include: "Handbook of Commonly Used Chinese Medicine Names and Alternative Names", "Standardized Terminology for Common Symptoms in Chinese Medicine Clinical Practice", "Comprehensive Dictionary of Chinese Medicine", and "Terminology for Clinical Diagnosis and Treatment in Chinese Medicine". Based on the high-frequency translated words and annotations in existing translations of ancient Chinese medicine books, as well as ancient Chinese words and their translations and classical Chinese function words and their translations found online, a comparison table of ancient and modern word meanings is constructed. The comparison table includes: terms in a dictionary of terms in the field of traditional Chinese medicine and fixed vocabulary in the field of traditional Chinese medicine. Classical Chinese words requiring polysemous word modification are included in the rule table. The translation rules for polysemous words are as follows: if a keyword appears in the preceding / following character of the classical Chinese word to be translated, then the translation is performed according to the modern Chinese word in the corresponding row of the keyword; if no keyword appears in the preceding / following character of the classical Chinese word, then the translation is performed according to the definition of the classical Chinese word in the ancient and modern word meaning comparison table; the keywords are derived from the original corpus. A custom TCM terminology database was constructed based on the aforementioned TCM terminology dictionary, book titles, personal names, dynasty names, and commonly used words. The book titles, personal names, and dynasty names were collected from the internet. The commonly used words were obtained by crawling Baidu Dictionary using BeautifulSoup, and the crawled content consisted of related words of Chinese characters that required partial modification.

3. The method according to claim 1, characterized in that, Based on the ancient and modern word meaning comparison table, the rule table, the custom terminology database, and the dictionary of terms in the field of traditional Chinese medicine, a rule-based method is used to translate the text to be segmented from classical Chinese to modern Chinese, resulting in the translated text, including: The terminology dictionary in the field of traditional Chinese medicine is sorted from longest to shortest to obtain a sorted terminology dictionary in the field of traditional Chinese medicine. Extract the words to be translated and their corresponding position information from the text to be segmented using the ancient and modern word meaning comparison table and the sorted dictionary of terms in the field of traditional Chinese medicine; The extracted words that need to be translated are translated according to the ancient and modern word meaning comparison table, the rule table, and the custom terminology database to obtain the translated text.

4. The method according to claim 3, characterized in that, The extracted words requiring translation are translated based on the ancient and modern word meaning comparison table, the rule table, and the custom terminology database to obtain the translated text, including: Based on the ancient and modern word meaning comparison table, a preliminary selection of words that need to be translated in classical Chinese is made. Based on the custom terminology database, determine whether the words to be translated need to be translated, and obtain the judgment result; If the judgment result is that translation is not required, then no translation will be performed; If the determination result indicates that translation is required, and the extracted words to be translated are words that need to be modified, if it is a polysemous word modification, then the extracted words to be translated are translated according to the rule table and the ancient and modern word meaning comparison table to obtain the translated text; if it is a partial modification, then words not in the custom terminology database are modified to obtain the translated text.

5. A word segmentation device for ancient Chinese medicine books, characterized in that, The apparatus comprising the method according to any one of claims 1 to 4, wherein the apparatus includes: The data acquisition module is used to acquire the text to be segmented from ancient Chinese medicine books; The translation data table and rule building module is used to construct ancient and modern word meaning comparison tables, rule tables, custom terminology databases, and a terminology dictionary in the field of traditional Chinese medicine. The local translation module is used to translate the text to be segmented from classical Chinese to modern Chinese using a rule-based method, based on the ancient and modern word meaning comparison table, the rule table, the custom terminology database, and the dictionary of terms in the field of traditional Chinese medicine, to obtain the translated text. The translated text includes: translation information of the translated word, original position information of the translated word, and the serial number of the translated word in the ancient and modern word meaning comparison table. The word segmentation module is used to segment the translated text with the aid of the Jieba word segmentation tool using the terminology dictionary in the field of traditional Chinese medicine, and then use the hidden Markov model to solve the out-of-vocabulary word problem to obtain the word segmentation result. The restoration module is used to translate the word segmentation results into the original text according to the ancient and modern word meaning comparison table and the rule table, so as to obtain the final word segmentation text.

6. The apparatus according to claim 5, characterized in that, The translation data table and rule-building module are also used to collect terms related to Chinese medicine, prescriptions, syndromes, diseases, symptoms, treatments, drug processing methods, and the human body from ancient Chinese medicine books. Based on the collected terms, a terminology dictionary for the field of Chinese medicine is constructed. These ancient Chinese medicine books include: *Handbook of Commonly Used Chinese Medicine Names and Alternative Names*, *Standardized Terminology for Common Symptoms in Traditional Chinese Medicine Clinical Practice*, *Comprehensive Dictionary of Traditional Chinese Medicine*, and *Terminology for Clinical Diagnosis and Treatment in Traditional Chinese Medicine*. Based on high-frequency translated words and annotations in existing translations of ancient Chinese medicine books, as well as classical Chinese words and their translations and classical Chinese function words and their translations found online, a glossary of ancient and modern meanings is constructed. This glossary includes terms from the terminology dictionary for the field of Chinese medicine and terms related to traditional Chinese medicine. Fixed vocabulary in the domain; Classical Chinese words requiring polysemous modification are included in the rule table. The translation rules for polysemous words are as follows: if a keyword appears in the preceding / following character of the Classical Chinese word to be translated, it is translated according to the modern Chinese in the corresponding row; if the keyword does not appear in the preceding / following character of the Classical Chinese word, it is translated according to the definition of the Classical Chinese word in the ancient and modern word meaning comparison table; the keywords are derived from the original corpus; a custom TCM terminology database is constructed based on the TCM terminology dictionary, book titles, personal names, dynasty names, and commonly used words; the book titles, personal names, and dynasty names are collected from the Internet; the commonly used words are obtained by crawling Baidu Dictionary using BeautifulSoup, and the crawled content is the related words of the Chinese characters that need to be partially modified.

7. The apparatus according to claim 5, characterized in that, The local translation module is also used to sort the terminology strings in the TCM terminology dictionary from longest to shortest to obtain a sorted TCM terminology dictionary; extract the words to be translated and their corresponding position information from the text to be segmented based on the ancient and modern word meaning comparison table and the sorted TCM terminology dictionary; and translate the extracted words to be translated based on the ancient and modern word meaning comparison table, the rule table, and the custom terminology library to obtain the translated text. The translated text includes: translation information of the translated word, original location information of the translated word, and the serial number of the translated word in the ancient and modern word meaning comparison table.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 4.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 4.

Images