Method, system and medium for constructing a variant character dictionary of ancient chinese medical books and text alignment

Abstract

The present application belongs to the technical field of natural language processing for traditional Chinese medicine ancient books, and particularly relates to a method and system for constructing a variant character dictionary and text alignment of traditional Chinese medicine ancient books, and a medium. The present application combines the recognition of variant characters and the construction of a variant character dictionary to achieve a text alignment method for traditional Chinese medicine ancient books. Specifically, the present application uses deep learning and natural language processing technology to automatically extract variant character features, significantly improving the coverage range and recognition accuracy; through dynamic programming, semantic similarity calculation and knowledge graph fusion, the multi-modal features are comprehensively considered to significantly improve the alignment accuracy. At the same time, the model can dynamically adapt to new texts and variant characters, and has stronger expansibility and adaptability; and the knowledge graph is used to optimize the alignment result, improving the accuracy and efficiency of text processing. The final generated result is the aligned text sequence, in which the variant characters are correctly recognized and mapped to standard characters. The present application has good application prospects in the digitization of traditional Chinese medicine ancient books.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of natural language processing for traditional Chinese medicine ancient books, and particularly relates to a method and system for constructing a variant character dictionary and text alignment of traditional Chinese medicine ancient books, and a medium. BACKGROUND

[0002] Traditional Chinese medicine ancient books are treasures of traditional Chinese culture, carrying rich medical knowledge and historical information accumulated over thousands of years. These ancient books are not only an important basis for the study of traditional Chinese medicine theory, but also a valuable resource for the clinical practice of traditional Chinese medicine. However, due to historical evolution, regional differences, and different writing habits, there are a large number of variant characters in traditional Chinese medicine ancient books, which has brought great challenges to the digitization, in-depth study, and widespread dissemination of ancient books.

[0003] Variant characters are different forms of the same character written by different writers at different historical periods or in different regions. They differ in form but are completely identical or similar in meaning. In traditional Chinese medicine ancient books, the existence of variant characters not only increases the difficulty of reading, but also may lead to misinterpretation and omission of information, thereby affecting the accurate understanding and grasp of the content of ancient books.

[0004] Therefore, the identification of variant characters and the text alignment of different versions of traditional Chinese medicine ancient books are important work contents of the study of traditional Chinese medicine ancient books. Currently, there have been some attempts in the prior art to use artificial intelligence and other technologies to perform such work. For example, patent CN116416630A "Method for detecting and recognizing Chinese variant characters" proposes a method for detecting and recognizing Chinese variant characters in ancient book images, which realizes accurate positioning, segmentation, and recognition of variant characters through deep learning technology. Patent CN116484019A "Method for constructing and outputting Chinese variant character dictionary library" provides a systematic method for constructing a Chinese variant character dictionary library, which collects variant character samples from authoritative materials such as "Chinese Dictionary" and stores their related information (such as four-corner number, character shape, and pinyin) in a MongoDB database, realizing efficient management and retrieval of variant characters. Patent CN118675190A "Method for digitizing Chinese ancient book images" proposes a complete method for digitizing Chinese ancient book images, including text detection, recognition, and error correction. In addition, the method also balances the frequency of appearance of rare characters and variant characters through data synthesis technology, improving the recognition ability of these special characters.

[0005] However, these prior arts still fail to solve various difficulties in the recognition of variant characters in traditional Chinese medical books and the text alignment work, including: 1. The existing variant character recognition technology is mostly focused on single recognition task or the updating mechanism is not perfect, which cannot provide a dynamically updated variant character dictionary library, so that the accuracy of subsequent variant character recognition and text alignment tasks is limited; 2. The existing technology is not optimized for special medical terms and ancient Chinese usage in traditional Chinese medical books; 3. There is no method in the existing technology to combine the variant character recognition results or the variant character dictionary with the text alignment task of different versions of traditional Chinese medical books, and there are technical obstacles in auxiliary text alignment according to the variant character recognition results or the variant character dictionary. SUMMARY

[0006] In view of the problems of the prior art, the present application provides a method and system for constructing a variant character dictionary of traditional Chinese medical books and text alignment, and a medium.

[0007] A method for constructing a variant character dictionary of traditional Chinese medical books comprises the following steps:

[0008] Step 1: input the original traditional Chinese medical book text data, and perform preprocessing to obtain character sequences and character images;

[0009] Step 2: adopt a convolutional neural network model to extract morphological features of characters from the character images; adopt a recurrent neural network to capture context information of the character sequences;

[0010] Step 3: based on the morphological features and the context information, recognize variant characters and their corresponding standard characters;

[0011] Step 4: construct the recognized variant characters and their corresponding standard characters into a variant character dictionary.

[0012] Preferably, in step 1, the preprocessing comprises at least one of the following operations: character coding unification, denoising processing, and text segmentation.

[0013] Preferably, the convolutional neural network model comprises an input layer, a convolutional layer, a pooling layer, and an output layer; wherein the input size of the input layer is 32x32 pixels, the convolution kernel of the convolutional layer is 3x3, and the pooling window size of the pooling layer is 2x2.

[0014] And / or, the recurrent neural network is an LSTM model.

[0015] Preferably, in step 2, an attention mechanism is introduced into the convolutional neural network model and / or the recurrent neural network to dynamically adjust the weight of the variant characters, and the calculation formula of the attention score is:

[0016]

[0017] wherein,e i attention score of the i j attention score of the j attention score of the

[0018] Preferably, in step 3, the feature vectors of the form features and context information are fused to obtain a comprehensive feature vector, and the comprehensive feature vector is classified by a Softmax classifier to identify the variant character and its corresponding standard character.

[0019] Preferably, for the variant characters already in the variant character dictionary, if it is found that the corresponding standard character has a more accurate match, the variant character dictionary is updated by an updating algorithm or an artificial review mechanism.

[0020] The updating algorithm comprises the following steps:

[0021] Step a, similarity calculation and matching: traverse the standard characters in the variant character dictionary, calculate the similarity of the feature vectors of the new identified variant characters and the standard characters; for each new identified variant character, select the standard character with the highest similarity as the optimal match; if the similarity of the optimal match is higher than a set threshold, it is considered that a more accurate match is found; otherwise, the original dictionary matching relationship is retained.

[0022] Step b, dictionary updating: check whether the new identified variant character exists in the dictionary; if not, add the new identified variant character and its optimal matching standard character to the variant character dictionary; if it exists, compare the similarity of the new matching standard character and the original standard character, if the new matching similarity is higher, update the dictionary, otherwise, retain the original matching.

[0023] The application also provides a traditional Chinese medical ancient book variant character dictionary construction system, comprising:

[0024] An input module configured to input original traditional Chinese medical ancient book text data;

[0025] A preprocessing module configured to preprocess the original traditional Chinese medical ancient book text data to obtain character sequences and character images;

[0026] A feature extraction module configured to extract form features of characters from the character images by using a convolutional neural network model, and capture context information of the character sequences by using a recurrent neural network;

[0027] A variant character identification module configured to identify variant characters and their corresponding standard characters based on the form features and the context information;

[0028] A variant character dictionary construction module configured to construct the identified variant characters and their corresponding standard characters into a variant character dictionary.​

[0029] This invention also provides a text alignment method for aligning texts in different versions of ancient Chinese medicine texts, comprising the following steps:

[0030] Step A: Construct a dictionary of variant characters according to the above method for constructing a dictionary of variant characters in ancient Chinese medical texts, and obtain a preliminarily aligned text sequence;

[0031] Step B involves aligning the text using an edit distance algorithm to obtain an optimized alignment result.

[0032] Step C: Perform text alignment using semantic similarity calculation to obtain optimized alignment results;

[0033] Step D: Construct a knowledge graph of TCM terminology and optimize the alignment results using semantic associations within the TCM terminology knowledge graph.

[0034] This invention also provides a text alignment system for text alignment of different versions of ancient Chinese medicine books, including:

[0035] The preliminary alignment module integrates the aforementioned system for constructing a dictionary of variant characters in traditional Chinese medicine, and obtains a text sequence after preliminary alignment.

[0036] The first optimization module is configured to perform text alignment using an edit distance algorithm to obtain an optimized alignment result;

[0037] The second optimization module is configured to perform text alignment through semantic similarity calculation to obtain optimized alignment results;

[0038] The third optimization module is configured to optimize the alignment results by utilizing semantic associations in the TCM terminology knowledge graph.

[0039] The present invention also provides a computer-readable storage medium storing: a computer program for implementing the above-described method for constructing a dictionary of variant characters in ancient Chinese medical texts, or a computer program for implementing the above-described text alignment method.

[0040] This invention provides a method for dynamically constructing a dictionary of variant characters in ancient Chinese medical texts based on machine learning and natural language processing technologies, as well as an automatic alignment technology for multiple versions of ancient Chinese medical texts. The technical solution of this invention has the following beneficial technical effects:

[0041] 1. The variant character recognition method of the present invention combines deep learning, natural language processing technology and attention mechanism to automatically extract variant character features without the need for manual character list compilation, which significantly improves the coverage and recognition accuracy, and has the advantages of high efficiency and accuracy.

[0042] 2. In the text alignment task, by using dynamic programming, semantic similarity calculation and knowledge graph fusion, multimodal features are comprehensively considered, which significantly improves the alignment accuracy.

[0043] 3. In the preferred scheme, the variant character dictionary can be updated, enabling the model to dynamically adapt to new texts and variant characters, thus having stronger scalability and adaptability, and effectively solving the shortcomings of the existing technology.

[0044] 4. In the preferred embodiment, the present invention incorporates knowledge graphs, making the method more applicable to ancient books in the field of traditional Chinese medicine.

[0045] 5. This invention is the first to realize a method that combines a dictionary of variant characters with multiple versions of ancient Chinese medicine books for alignment tasks, which improves the degree of automation of the task, reduces the need for manual intervention, and is suitable for the digital processing of massive amounts of ancient books.

[0046] Obviously, based on the above description of the present invention, and according to common technical knowledge and conventional methods in the field, various other modifications, substitutions, or alterations can be made without departing from the basic technical concept of the present invention.

[0047] The following detailed embodiments further illustrate the above-described content of the present invention. However, this should not be construed as limiting the scope of the present invention to the following examples. All technologies implemented based on the above-described content of the present invention fall within the scope of the present invention. Attached Figure Description

[0048] Figure 1 This is a schematic diagram of the method flow according to an embodiment of the present invention.

[0049] Figure 2 The results are shown in Figure 1, which is a verification of the text processing of a certain ancient Chinese medicine book.

[0050] Figure 3 The result is shown in Figure 2, which is a verification result using a section of ancient Chinese medicine text as an example.

[0051] Figure 4 To verify the accuracy and loss curve of the training process, a specific section of ancient Chinese medicine text was used as an example.

[0052] Figure 5 To verify the statistical results of the occurrence of variant characters, a certain section of ancient Chinese medicine text was processed as an example. Detailed Implementation

[0053] It should be noted that the algorithms for data acquisition, transmission, storage and processing steps not specifically described in the embodiments, as well as the hardware structures and circuit connections not specifically described, can all be implemented using content already disclosed in the prior art.

[0054] Example 1: Method and System for Constructing a Dictionary of Variant Characters in Ancient Chinese Medicine Texts

[0055] The system in this embodiment includes:

[0056] The input module is configured to accept raw text data from ancient Chinese medicine texts.

[0057] The preprocessing module is configured to preprocess the original ancient Chinese medicine text data to obtain character sequences and character images;

[0058] The feature extraction module is configured to extract the morphological features of characters from character images using a convolutional neural network model; and to capture the contextual information of character sequences using a recurrent neural network.

[0059] The variant character recognition module is configured to recognize variant characters and their corresponding standard characters based on the morphological features and context information.

[0060] The variant character dictionary construction module is configured to construct a variant character dictionary from the identified variant characters and their corresponding standard characters.

[0061] The method for constructing a dictionary of variant characters in ancient Chinese medical texts using the above system includes the following steps (such as...). Figure 1 (as shown)

[0062] Step 1: Input the original ancient Chinese medicine text data, perform preprocessing, and obtain character sequences and character images.

[0063] 1. Collection of original ancient Chinese medicine text data:

[0064] Collect textual data from different versions of ancient Chinese medicine books, including scanned versions (such as PDF and image files) and electronic texts (such as TXT and XML).

[0065] Data sources include digitized ancient books from libraries, publicly available ancient book databases, and text resources provided by academic institutions.

[0066] 2. Pretreatment:

[0067] This step requires inputting the original text data of ancient Chinese medicine books.

[0068] The preprocessing steps include character encoding standardization, noise reduction, and text segmentation.

[0069] Character encoding standardization: Convert all text to UTF-8 encoding to ensure character consistency. This step is for subsequent text processing, especially as input to the LSTM model, because LSTM needs to process character sequences, and a standardized encoding format can avoid character parsing errors.

[0070] Denoising: Use regular expressions to remove irrelevant punctuation marks, whitespace characters, and noisy characters generated during optical character recognition (OCR).

[0071] Text segmentation: The text is segmented based on a Traditional Chinese Medicine (TCM) terminology dictionary to ensure the integrity of the terminology. For scanned TCM ancient texts, OCR technology is used to convert them into electronic text while extracting the image information of each character and uniformly adjusting its size to 32×32 pixels, which is then used as input data for the CNN model. Based on the term length distribution in the TCM terminology dictionary, a reasonable character sequence length is determined, such as taking the average term length plus a certain upper and lower fluctuation range. The character sequences in the text are then truncated or padded according to this length to meet the input requirements of the LSTM model.

[0072] The output data of this step is: preprocessed text data, including character sequences (flowing to LSTM) and character images (flowing to CNN).

[0073] Step 2: Use a convolutional neural network model to extract the morphological features of characters from the character image; use a recurrent neural network to capture the contextual information of the character sequence.

[0074] The input data for this step includes the character images input to the CNN model and the character sequences input to the LSTM model.

[0075] 1. Model selection:

[0076] CNN Model: Uses Convolutional Neural Networks (CNNs) to extract morphological features of characters. CNNs capture local features and global structure of characters through convolutional layers and pooling layers.

[0077] LSTM model: Uses recurrent neural networks (RNN) and their variant LSTM to capture contextual information of character sequences.

[0078] The specific implementations of the two models are as follows:

[0079] (1) CNN model:

[0080] Input layer: Character image (32×32 pixels).

[0081] Convolutional layer: Features are extracted using 3×3 convolutional kernels, and the activation function is ReLU.

[0082] Pooling layer: Use a 2×2 max pooling layer to reduce feature dimensionality.

[0083] Output layer: Feature vectors represent the morphological features of characters. These feature vectors serve as input to the classifier, helping to identify variant characters. The feature vectors output by the CNN model not only contain the morphological features of characters but also learn the mapping relationship between feature vectors and standard characters through comparison with labeled standard character samples. This provides effective feature representation and category label information for training the Softmax classifier. The dimension of the output feature vector of the CNN model is...dCNN This will be adjusted to match the input dimension of the Softmax classifier through a fully connected layer. dsoftmax The matching vector.

[0084] (2) LSTM model:

[0085] Input data: The input to the LSTM model is a sequence of characters encoded in UTF-8. Each character is converted into a fixed-length vector representation through an embedding layer. The embedding layer has a dimension of [missing value]. The weights of the embedding layer can be updated through backpropagation during training to learn more efficient character vector representations.

[0086] Forget gate: The forget gate determines which information is discarded from the cell state. The output is represented as:

[0087]

[0088] in, Output for the forget gate. This is the weight matrix. σ is the bias term, and σ is the sigmoid activation function.

[0089] Input gate: The input gate determines which new information will be written to the cell state. The output is represented as:

[0090]

[0091] in, For input gate output, These are candidate memory units. and These are the weight matrices for the input gate and candidate memory units, respectively. and These are the bias terms for the input gate and the candidate memory unit, respectively.

[0092] Cell state update: The cell state update combines information from the forget gate and the input gate. The update formula is:

[0093]

[0094] Output gate: The output gate determines which parts of the cell state will be output. The output is represented as:

[0095]

[0096] The final hidden state is , representing the output information at the current time step. During text alignment, the matching cost between characters can be adjusted based on the hidden states of the LSTM model. For example, when the LSTM hidden states of two characters are semantically similar, their matching cost can be reduced. At the same time, the weight matrix and bias terms in the LSTM model can be fine-tuned based on the accuracy and efficiency of the alignment results to further optimize the performance of the alignment algorithm.

[0097] This step outputs the extracted features.

[0098] (3) Optimization method:

[0099] An attention mechanism is introduced to dynamically adjust the weight of variant characters. The formula for calculating the attention score is as follows:

[0100]

[0101] in, ei For the first i Attention score for each variant character, ej For the first j Attention scores for each variant character. The advantage of dynamically adjusting weights is that they can be dynamically assigned based on the importance of variant characters in the context, improving recognition accuracy and robustness. During classification, the feature vector corresponding to each variant character is weighted according to its attention score; variant characters with higher weights have a greater influence on the classification decision. In this way, the model can focus more on variant characters that are more important in the context, thereby improving recognition accuracy and robustness. During training, the parameters of the attention mechanism are learned by optimizing the algorithm, so that the attention scores can better reflect the importance of variant characters in the context.

[0102] This embodiment introduces attention mechanisms into both CNN and LSTM. In CNN, the attention mechanism can focus on key parts of the character image (such as strokes and structure), enhancing the accuracy of morphological feature extraction. In LSTM, the attention mechanism can dynamically focus on important contextual information in the character sequence, improving the understanding of character semantics and context. This combination allows the model to dynamically adjust weights based on the features of the input data, handling different characters and contexts more flexibly, thereby improving the accuracy of variant character recognition.

[0103] Step 3: Based on the morphological features and contextual information, identify variant characters and their corresponding standard characters.

[0104] The input data for this step is the features extracted by CNN and LSTM models.

[0105] A feature fusion approach is employed, concatenating or weighting the character morphological feature vector output by the CNN with the contextual information feature vector output by the LSTM to form a comprehensive feature vector. This comprehensive feature vector is then input into a Softmax classifier to calculate the probability that each variant character belongs to a given standard character, thereby completing the recognition process.

[0106] Classifier training:

[0107] A feature fusion approach is used, concatenating or weighting the character morphological feature vector output by the CNN with the contextual information feature vector output by the LSTM to form a comprehensive feature vector. This comprehensive feature vector is then input into a Softmax classifier to calculate the probability that each variant character belongs to a given standard character, thus completing the recognition process. The feature vector output by the CNN model has a dimension of [missing information]. dCNN The input dimension is adjusted to match that of the Softmax classifier through a fully connected layer. dsoftmax The matching vectors. The final output of the LSTM model is the final hidden state. ht .

[0108] The loss function used is cross-entropy loss:

[0109]

[0110] in, For real labels, To predict probabilities, during training, the Adam optimizer is used to update the parameters of the CNN model, LSTM model, and Softmax classifier based on the cross-entropy loss function. This allows the model to better learn the mapping relationship between feature vectors and standard characters, thereby improving the accuracy of variant character recognition.

[0111] Step 4: Construct a dictionary of variant characters by identifying the variant characters and their corresponding standard characters.

[0112] The identified variant characters and their corresponding standard characters are stored in a dictionary, with the following dictionary structure:

[0113] Dictionary = {Variant character 1: Standard character 1, Variant character 2: Standard character 2, ...}

[0114] If a more accurate match is found for a standard character that already exists in the dictionary, the dictionary can be updated through algorithm updates or manual review mechanisms to ensure its accuracy and timeliness.

[0115] The update algorithm specifically includes the following steps:

[0116] 1. Similarity Calculation and Matching: Traverse the standard characters in the variant character dictionary and calculate the similarity (e.g., cosine similarity) between their feature vectors and the newly identified variant characters. For each newly identified variant character, select the standard character with the highest similarity as the optimal match. If the similarity of the optimal match is higher than a set threshold (e.g., 0.9), it is considered that a more accurate match has been found; otherwise, the original dictionary matching relationship is retained.

[0117] 2. Dictionary Update: Check if the newly identified variant character already exists in the dictionary. If it does not exist, add the new variant character and its best matching standard character to the variant character dictionary; if it already exists, compare the similarity between the new matching standard character and the original standard character. If the new match has a higher similarity, update the dictionary; otherwise, retain the original match.

[0118] The update algorithm periodically validates the variant character dictionary with new text data, checks for incorrect matches or omissions, and fine-tunes the dictionary based on the validation results, or adjusts the similarity threshold and optimizes the feature extraction model. The updated variant character dictionary, similarity threshold, or feature extraction model is then applied to subsequent text processing tasks, forming a dynamic update loop that ensures the dictionary's accuracy and timeliness.

[0119] Example 2: Method and System for Aligning Ancient Chinese Medicine Texts

[0120] The system in this embodiment includes:

[0121] The preliminary alignment module integrates the traditional Chinese medicine variant character dictionary construction system described in Example 1, and obtains the text sequence after preliminary alignment.

[0122] The first optimization module is configured to perform text alignment using an edit distance algorithm to obtain an optimized alignment result;

[0123] The second optimization module is configured to perform text alignment through semantic similarity calculation to obtain optimized alignment results;

[0124] The third optimization module is configured to optimize the alignment results by utilizing semantic associations in the TCM terminology knowledge graph.

[0125] The method for aligning ancient Chinese medical texts using the above system includes the following steps (such as...). Figure 1 (as shown)

[0126] Step A: Construct a variant character dictionary according to the method for constructing a TCM variant character dictionary in Example 1, obtaining a preliminarily aligned text sequence. The text sequence obtained after completing variant character recognition and dynamic dictionary updates is the preliminarily aligned text sequence, in which variant characters have been correctly identified and mapped to standard characters, thereby improving the accuracy and consistency of alignment. Based on this, subsequent steps (such as dynamic programming alignment and context optimization) will further optimize the alignment results, combining semantic similarity calculation and knowledge graph fusion to further improve the accuracy and efficiency of alignment.

[0127] Step B involves aligning the text using an edit distance algorithm to obtain an optimized alignment result.

[0128] This step uses the edit distance algorithm to calculate the similarity between two text sequences. The text sequences are derived from the initially aligned text sequences. These text sequences may contain variant characters; by using the correspondence in a variant character dictionary, variant characters can be mapped to standard characters, thereby improving the accuracy of text alignment. Before performing text alignment, the two text sequences are preprocessed. Natural language processing techniques such as word segmentation, part-of-speech tagging, and dependency parsing are used to determine the structural features of the text sequences. Then, based on these structural features, the start and end points of alignment are determined, and the edit distance algorithm is used within this range to perform alignment, thereby improving the accuracy and efficiency of alignment.

[0129]

[0130] in, To calculate the position of two text sequences i Arrive at the location j Edit distance refers to the minimum number of single-character editing operations required to transform one string into another, including operations such as insertion, deletion, and replacement.

[0131]

[0132] Among them, Cost ( xi,yj ) is a character xi and yj The matching cost represents the similarity between two characters. Character pairs with lower matching costs are more likely to be related as standard and variant characters. The data in this step is closely related to the output data from previous steps (such as the correspondence between variant and standard characters in the dictionary) and is used to optimize the alignment results.

[0133] The output data for this step is: a text sequence aligned by dynamic programming.

[0134] Step C involves aligning text using semantic similarity calculation to obtain optimized alignment results.

[0135] By combining semantic similarity calculation, semantic vectors of characters (such as Word2Vec or BERT embeddings) are used to calculate the semantic similarity between two characters. This helps to handle semantic variations of characters in different contexts and improves alignment accuracy. Pre-trained Word2Vec or BERT models are used to embed characters in the text, obtaining the semantic vector for each character. When calculating the matching cost between characters, their semantic similarity is considered; for example, when the semantic vectors of two characters are similar, their matching cost is reduced, thus making the alignment result more consistent with semantic logic.

[0136]

[0137] in, For semantic similarity, Embedding( x ) and Embedding(y) represent characters respectively. x and y The semantic vector.

[0138] The matching cost adjustment formula is:

[0139]

[0140] in, α This is the weighting coefficient for semantic similarity, which can be adjusted based on experimental results.

[0141] This step outputs the following data: a text sequence optimized for context.

[0142] Step D: Construct a knowledge graph of TCM terminology and optimize the alignment results using semantic associations within the TCM terminology knowledge graph.

[0143] A knowledge graph of TCM terminology is constructed, and the semantic relationships within the graph are used to optimize the alignment results. During the knowledge graph fusion process, the existing correspondences between variant characters and standard characters in the variant character dictionary can be used to further optimize the alignment results and ensure correct matching between variant characters and standard characters.

[0144] By mining the terminology, concepts, and relationships within ancient Chinese medical texts, a knowledge graph of Chinese medical terminology is constructed. When calculating the matching cost between characters, their semantic association weights within the knowledge graph are considered. When two characters have a strong semantic association in the knowledge graph, their matching cost is reduced, thus making the alignment results more consistent with the semantic logic of Chinese medical terminology. Semantic associations in the knowledge graph can be represented by edge weights; higher edge weights indicate a stronger semantic association between two terms.

[0145] The matching cost adjustment formula is:

[0146]

[0147] in For characters and Semantic association weights in knowledge graphs β These are the weight coefficients for semantic association weights.

[0148] This step outputs the final aligned text sequence.

[0149] The following example, using a passage from an ancient Chinese medicine text, illustrates the actual operational effect.

[0150] The specific method for aligning ancient Chinese medical texts was implemented according to Example 2, and the results are as follows:

[0151] Examples of text processing results for ancient Chinese medicine books are as follows: Figure 2 , 3 As shown, Figure 2 Includes:

[0152] File encoding detection results, preprocessed text comparison, and variant character extraction results. Figure 5 This is a visual representation of the results of variant character extraction.

[0153] Figure 3 The diagram illustrates the model training process and performance. Figure 4 This is a visual representation of the accuracy and loss of this operation. Figure 3 The display shows that the variant character dictionary has been built, indicating that the dictionary construction is complete and a preliminary text alignment sequence has been formed, providing preliminary data for subsequent operations.

[0154] As can be seen from the above example, the system processing result of this embodiment is as follows:

[0155] 1. Data preprocessing results

[0156] After preprocessing, noisy characters in the ancient texts were effectively removed, and the character encoding was unified to UTF-8 format, significantly improving the text quality and providing a good foundation for subsequent feature extraction and model training.

[0157] 2. Feature extraction results

[0158] The CNN model successfully extracted the feature vectors of variant characters. Each variant character corresponds to a 32×32 pixel image feature, and the feature vector has a dimension of 128, which can effectively represent the shape and structural information of variant characters.

[0159] The LSTM model successfully extracted the contextual feature vectors of variant characters. Each variant character corresponds to a contextual feature based on the character sequence. The dimension of the feature vector depends on the number of hidden units in the LSTM layer, which can effectively represent the semantic and contextual information of variant characters in the text sequence.

[0160] 3. Results of variant character recognition and dictionary construction

[0161] The model achieved an accuracy of over 95% on the test set, demonstrating its high ability to recognize variant characters. The constructed dictionary of variant characters accurately maps variant characters in ancient texts to their corresponding modern Chinese characters, providing crucial support for reading and understanding ancient texts.

[0162] 4. Text alignment results

[0163] The high alignment similarity scores between the two ancient texts indicate a high degree of similarity in content. The alignment results can effectively assist in the comparison and proofreading of different versions of ancient texts, ensuring consistency in content across various editions.

[0164] As can be seen from the above embodiments, this invention is the first to realize a method and system for combining a dictionary of variant characters with the task of aligning ancient Chinese medical texts. It can efficiently and accurately perform the task of aligning ancient Chinese medical texts and has excellent application prospects.

Claims

1. A method for constructing a dictionary of variant characters in ancient Chinese medical texts and combining it with text alignment, characterized in that... Text alignment for different versions of ancient Chinese medical texts includes the following steps: Step A: Construct a dictionary of variant characters according to the method for constructing a dictionary of variant characters in ancient Chinese medicine texts, and obtain a preliminarily aligned text sequence; The method for constructing a dictionary of variant characters in ancient Chinese medical texts includes the following steps: Step 1: Input the original ancient Chinese medicine text data, perform preprocessing, and obtain character sequences and character images; Step 2: Extract morphological features of characters from character images using a convolutional neural network (CNN) model; capture contextual information of character sequences using a recurrent neural network (RNN); the CNN model includes an input layer, a convolutional layer, a pooling layer, and an output layer; wherein the input size of the input layer is 32×32 pixels, the convolutional kernel of the convolutional layer is 3×3, and the pooling window size of the pooling layer is 2×2; the RNN is an LSTM model. Step 3: Based on the morphological features and contextual information, identify variant characters and their corresponding standard characters; Step 4: Construct a variant character dictionary by identifying the variant characters and their corresponding standard characters; for variant characters already in the variant character dictionary, if a more accurate match is found in the corresponding standard character, the variant character dictionary is updated using an update algorithm. The update algorithm includes the following steps: Step a, Similarity calculation and matching: Traverse the standard characters in the variant character dictionary and calculate the similarity between their feature vectors and the newly identified variant characters; for each newly identified variant character, select the standard character with the highest similarity as the optimal match; if the similarity of the optimal match is higher than a set threshold, it is considered that a more accurate match has been found; otherwise, the original dictionary matching relationship is retained. Step b, dictionary update: Check if the newly identified variant character already exists in the dictionary; if not, add the newly identified variant character and its best matching standard character to the variant character dictionary; if it already exists, compare the similarity between the new matching standard character and the original standard character. If the new matching has a higher similarity, update the dictionary; otherwise, retain the original matching. The text sequence obtained after completing variant character recognition and dynamic dictionary update is used as the initial aligned text sequence; Step B involves aligning the text using an edit distance algorithm to obtain an optimized alignment result. Alignment is performed using an edit distance algorithm: in, For two text sequences from position i Arrive at the location j Edit distance; Among them, Cost ( x i ,y j ) is a character x i and y j The matching cost; the matching cost represents the similarity between two characters; Step C: Perform text alignment using semantic similarity calculation to obtain optimized alignment results; When calculating the matching cost between characters, semantic similarity is considered. When the semantic vectors of two characters are similar, their matching cost is reduced. Step D: Construct a knowledge graph of TCM terminology and optimize the alignment results using semantic associations in the TCM terminology knowledge graph; When calculating the matching cost between characters, their semantic association weight in the knowledge graph is considered. When two characters have a strong semantic association in the knowledge graph, their matching cost is reduced.

2. The method for constructing a dictionary of variant characters in ancient Chinese medical texts and combining it with text alignment according to claim 1, characterized in that: In step 1, the preprocessing includes at least one of the following operations: character encoding unification, noise reduction, and text segmentation.

3. The method for constructing a dictionary of variant characters in ancient Chinese medical texts and combining it with text alignment according to claim 1, characterized in that: In step 2, an attention mechanism is introduced into the convolutional neural network model and / or recurrent neural network to dynamically adjust the weights of variant characters. The formula for calculating the attention score is: in, e i For the first i Attention score for each variant character, e j For the first j Attention score for each variant character.

4. The method for constructing a dictionary of variant characters in ancient Chinese medical texts and combining it with text alignment according to claim 1, characterized in that: In step 3, the feature vectors of the morphological features and contextual information are fused to obtain a comprehensive feature vector. The comprehensive feature vector is then classified using a Softmax classifier to identify variant characters and their corresponding standard characters.

5. A text alignment system, characterized in that, The method for constructing a dictionary of variant characters in ancient Chinese medical texts and combining it with text alignment, as described in any one of claims 1-4, is used for text alignment of different versions of ancient Chinese medical texts, comprising: The initial alignment module is configured to obtain a pre-aligned text sequence. The first optimization module is configured to perform text alignment using an edit distance algorithm to obtain an optimized alignment result; The second optimization module is configured to perform text alignment through semantic similarity calculation to obtain optimized alignment results; The third optimization module is configured to optimize the alignment results by utilizing semantic associations in the TCM terminology knowledge graph.

6. A computer-readable storage medium, characterized in that, It stores a computer program for implementing the method of constructing and aligning a dictionary of variant characters in ancient Chinese medical texts as described in any one of claims 1-4.

Images