A method and system for semantic understanding and segmentation of classical Chinese texts
By training and optimizing the Llama3.2 model, a separation model suitable for ancient Chinese was constructed, which solved the problem of word boundary and sentence feature recognition in ancient Chinese text processing, and realized the accuracy of word segmentation and part-of-speech labeling and multi-scene adaptability.
Patent Information
- Application Number
- CN202411963910.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-30
- Publication Date
- 2025-08-01
- Estimated Expiration
- 2044-12-30
AI Technical Summary
Existing NLP models such as BERT and GPT lack adaptability in ancient text processing, making it difficult to accurately identify the word boundaries and sentence characteristics of ancient texts, and the traditional word segmentation tools have low accuracy in ancient texts, so they cannot effectively deal with the diverse ancient text structure and semantics.
By collecting ancient text data, generating corpus data sets and preprocessing, the vocabulary of the Llama3.2 model is extended, sequence annotation is used using BIO annotation method, combining adaptive optimization algorithms and loss function adjustment models, optimizing ancient text word segmentation and part-of-speech annotation, and building a separation model suitable for ancient text.
It improves the accuracy of ancient Chinese word segmentation and the accuracy of part-of-speech labeling, is suitable for complex ancient Chinese grammatical structures, supports multi-scene application expansion, and improves the automation and intelligence level of ancient Chinese data processing.
Smart Images
Figure CN119886134B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of text processing, and particularly relates to a method and system for semantic understanding and segmentation of ancient Chinese text. Background Art
[0002] With the rapid development of artificial intelligence technology, natural language processing (NLP) models based on deep learning have made remarkable progress in text semantic understanding. However, most NLP models (such as BERT, GPT series, Llama) are mainly trained for modern languages and lack the adaptability to ancient Chinese text. Ancient Chinese has the characteristics of concise grammar, variable word order, and complex parts of speech. Traditional semantic understanding technologies have significant limitations in processing ancient Chinese. Therefore, new technical requirements are put forward for the semantic understanding of ancient Chinese text.
[0003] Currently, mainstream semantic understanding technologies (such as pre-trained models like BERT and GPT) perform well in modern text processing. They can capture semantic relationships through context encoding and achieve word segmentation, part-of-speech tagging, and semantic parsing. However, the training corpora of these models are mainly based on modern Chinese and English, lacking adaptability to the characteristics of ancient Chinese. Traditional Chinese word segmentation tools (such as Jieba) have limited performance in processing ancient Chinese, mainly due to the fuzzy word boundaries, flexible sentence-breaking rules, and the complexity of parts of speech and syntax in ancient Chinese. In addition, the feature extraction and grammar parsing methods for modern texts are usually based on fixed syntactic structures, while a large number of special syntactic structures such as ellipsis sentences and inverted sentences in ancient Chinese make it difficult for traditional models to adapt.
[0004] However, the existing technologies still have significant deficiencies in ancient Chinese processing. Pre-trained models (such as BERT and GPT) lack the understanding of unique vocabulary and grammar in ancient Chinese, and their adaptability after fine-tuning is limited, making it difficult to effectively handle polysemous words and complex grammar in ancient Chinese. The accuracy of word segmentation tools in ancient Chinese is relatively low, and it is difficult to accurately identify word boundaries and sentence pattern features. The scarcity of ancient Chinese corpora further limits the depth of model learning of ancient Chinese characteristics, resulting in poor word segmentation and part-of-speech tagging effects. In addition, conventional models have poor generalization ability in different types of ancient Chinese texts and cannot effectively handle diverse ancient Chinese structures and semantics, restricting their application scope. Summary of the Invention
[0005] The present invention aims to address the technical problems existing in the background art, and provides a method and system for semantic understanding and segmentation of ancient Chinese text, which can improve the accuracy of ancient Chinese word segmentation, achieve accurate part-of-speech tagging, be applicable to complex ancient Chinese grammar structures, support multi-scenario application expansion, and improve the automation and intelligence level of ancient Chinese data processing.
[0006] To achieve the above technical objectives, the present invention adopts the following technical solutions:
[0007] A method for semantic understanding and segmentation of ancient Chinese text, comprising the following steps:
[0008] S1. Collect ancient Chinese text data to generate an ancient Chinese corpus dataset;
[0009] S2. Preprocess the ancient Chinese corpus dataset to generate an annotated dataset, and divide the annotated dataset into a training set and a validation set;
[0010] S3. Input the training set into the Llama 3.2 model, train the Llama 3.2 model, and expand the ancient Chinese vocabulary of the Llama 3.2 model;
[0011] S4. Use the trained Llama 3.2 model as the base model, re-initialize the word embedding layer with the expanded Llama 3.2 model vocabulary to obtain an expanded Llama 3.2 model;
[0012] S5. Design the ancient Chinese word segmentation task and part-of-speech tagging task as sequence annotation, perform sequence annotation on the training set in step S2 to form an optimized ancient Chinese annotated corpus set;
[0013] S6. Use the optimized ancient Chinese annotated corpus set to train the expanded Llama 3.2 model obtained in step S4, and introduce a loss function to adjust the model to obtain an adjusted Llama 3.2 model;
[0014] S7. Use the validation set in step S2 to evaluate the adjusted Llama 3.2 model, and introduce a loss function to adjust the model again. Determine whether the difference between the predicted value of the model after the second adjustment and the actual value in the validation set meets the preset standard. If not, perform optimization;
[0015] S8. Use an adaptive optimization algorithm to optimize the parameters of the Llama 3.2 model whose predicted value does not meet the preset standard, and reduce the learning rate of the model. Denote the optimized model as the segmentation model;
[0016] S9. Input the ancient Chinese text into the segmentation model, perform ancient Chinese word segmentation and part-of-speech tagging on the ancient Chinese text, and output the ancient Chinese text result.
[0017] Preferably, in step S2, the specific content of preprocessing the ancient Chinese corpus dataset includes:
[0018] S201. Text cleaning: Remove punctuation marks, redundant characters, and incorrect content in the ancient Chinese corpus data, and unify the text format;
[0019] S202. Corpus annotation: Annotate the boundaries of ancient Chinese word segmentation and parts of speech for the ancient Chinese corpus dataset after text cleaning to generate an annotated dataset.
[0020] Preferably, in step S3, expanding the ancient Chinese vocabulary of the Llama 3.2 model includes adding the vocabulary and expressions in ancient Chinese to the vocabulary of the Llama 3.2 model.
[0021] Preferably, step S5 is specifically as follows: using the BIO annotation method to annotate the boundaries of ancient Chinese word segmentation, obtaining ancient Chinese word segmentation, and combining part-of-speech to form a joint part-of-speech annotation task, and performing sequence annotation on the training set to form an optimized ancient Chinese annotation corpus.
[0022] Preferably, in step S6, during the training process, minimizing the cross-entropy loss between the model prediction and the true annotation is used as the adjustment objective function, specifically:
[0023]
[0024] where x i represents the input ancient Chinese text, y i is the boundary and part-of-speech annotation label of the corresponding ancient Chinese word segmentation, f Llama represents the adjusted Llama 3.2 model, θ is the model parameter, and L is the cross-entropy loss function;
[0025]
[0026] where N is the number of samples in the training set, T is the sequence length of each sample, y ij is the true label at the j-th position of the i-th sample, is the probability distribution at the j-th position of the i-th sample predicted by the model.
[0027] Preferably, in step S7, the specific content of evaluating the adjusted Llama 3.2 model includes:
[0028] S701. Evaluation of the precision, recall, and F1 score of ancient Chinese word segmentation:
[0029] The formula for precision is expressed as follows:
[0030]
[0031] where P represents precision, TP represents the number of correctly predicted ancient Chinese word segmentation boundaries, and FP represents the number of boundaries predicted as ancient Chinese word segmentation boundaries but actually not;
[0032] The formula for recall is expressed as follows:
[0033]
[0034] where FN represents the number of true word segmentation boundaries that the adjusted Llama 3.2 model fails to predict;
[0035] The formula for the F1 score is expressed as follows:
[0036]
[0037] Among them, F1-Score represents the F1 score;
[0038] S702. Evaluation of the accuracy of part-of-speech tagging:
[0039] The formula for the accuracy of part-of-speech tagging is expressed as follows:
[0040]
[0041] Among them, Accuracy represents the accuracy of part-of-speech tagging, the number of correctly predicted tags represents the number of predicted part-of-speech tags that are consistent with the true part-of-speech tags, and the total number of tags represents the total number of all part-of-speech tags predicted using the validation set;
[0042] S703. Calculate the loss function to adjust the Llama3.2 model again, and the formula is expressed as follows:
[0043]
[0044] Among them, L' represents the average cross-entropy loss value, N' is the number of samples in the validation set, T' is the sequence length of each sample, C represents the number of part-of-speech categories, is the true label of the k-th category at the j'-th position of the i'-th sample, is the model prediction probability of the k-th category at the j'-th position of the i'-th sample.
[0045] Preferably, step S8 specifically includes:
[0046] S801. Parameter optimization: Use the AdamW optimizer to update the model parameters, and the formula is expressed as follows:
[0047]
[0048] Among them, η is the learning rate, θ t+1 is the parameter of the model at the (t + 1)-th iteration of the next iteration, θ t is the parameter of the model at the t-th iteration of the current iteration, and λ is the weight decay coefficient;
[0049] S802. Learning rate scheduling: Use the cosine annealing learning rate scheduler to decrease the learning rate of the model, and the formula is expressed as follows:
[0050]
[0051] Among them, η t is the current learning rate, ηmax is the maximum learning rate, η min is the minimum learning rate, t 次 is the current iteration number, T 次 is the total number of iterations.
[0052] Preferably, step S9 specifically includes:
[0053] S901. Input the classical Chinese text into the segmentation model. The segmentation model performs classical Chinese word segmentation on the classical Chinese text to generate a word segmentation result with word boundaries;
[0054] S902. Based on the word segmentation result with word boundaries, extract the part-of-speech of each word to generate a complete classical Chinese semantic structure including classical Chinese word segmentation and part-of-speech tagging;
[0055] S903. Combine the classical Chinese word segmentation and part-of-speech, and use the segmentation model to analyze the grammatical structure;
[0056] S904. Output the classical Chinese text information, and output the word segmentation result and part-of-speech tagging result of the classical Chinese text.
[0057] The present invention also provides a classical Chinese text semantic understanding and segmentation system, which uses the classical Chinese text semantic understanding and segmentation method as described above, including:
[0058] A data processing module, a model training and extension module, a data optimization module, an adjustment and evaluation module, a model optimization module, and an output module;
[0059] The data processing module is used to collect classical Chinese text data to generate a classical Chinese corpus dataset, preprocess the classical Chinese corpus dataset to obtain an annotated dataset, and divide the annotated dataset into a training set and a validation set;
[0060] The model training and extension module is used to train the Llama 3.2 model using the training set obtained by the data processing module, expand the classical Chinese vocabulary of the Llama 3.2 model, and use the trained Llama 3.2 model as a base model to re-initialize the word embedding layer using the expanded Llama 3.2 model vocabulary to obtain an extended Llama 3.2 model;
[0061] The data optimization module is used to perform sequence annotation on the training set obtained by the data processing module to form an optimized classical Chinese annotated corpus dataset;
[0062] The adjustment and evaluation module is used to train the extended Llama 3.2 model obtained by the model training and extension module with the optimized classical Chinese annotation corpus obtained by the data optimization module, and introduce a loss function to adjust the model to obtain an adjusted Llama 3.2 model. At the same time, the adjusted Llama 3.2 model is evaluated using the validation set obtained by the data processing module, and the loss function is introduced again to adjust the model to determine whether the difference between the predicted value of the model after the second adjustment and the actual value in the validation set meets the preset standard;
[0063] The model optimization module is used to optimize the parameters of the Llama 3.2 model whose predicted value obtained by the adjustment and evaluation module does not meet the preset standard, and reduce the learning rate of the model to generate a segmented model;
[0064] The output module is used to output the result of the input classical Chinese text. Using the segmented model obtained by the model optimization module, the classical Chinese text is segmented and part-of-speech tagged, and the result of the classical Chinese text is output.
[0065] Compared with the prior art, the beneficial effects produced by the present invention are as follows:
[0066] (1) By training, adjusting, evaluating and optimizing the Llama 3.2 model, the present invention constructs a segmented model suitable for classical Chinese word segmentation and part-of-speech tagging, improves the accuracy of classical Chinese word segmentation, realizes accurate part-of-speech tagging, is suitable for complex classical Chinese grammar structures, reduces the cost of classical Chinese corpus processing, and shows remarkable effects in improving the ability of classical Chinese semantic understanding and processing;
[0067] (2) The segmented model constructed by the present invention supports multi-scenario application expansion, can be applied to scenarios such as classical Chinese translation, classical Chinese knowledge graph construction, and classical Chinese search engines, improves the generalization ability of the model, enhances the automation and intelligence level of classical Chinese data processing, and provides technical support for the digitization and intelligence of classical Chinese. BRIEF DESCRIPTION OF THE DRAWINGS
[0068] Figure 1 It is a flowchart of the method for segmenting and understanding the semantics of classical Chinese text according to an embodiment of the present invention;
[0069] Figure 2 It is a technical roadmap of the method for segmenting and understanding the semantics of classical Chinese text according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION
[0070] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.
[0071] Embodiment 1
[0072] Combined with Figure 1 As shown, the embodiment of the present invention provides a method for separating the semantic understanding of ancient Chinese text, including the following steps:
[0073] S1. Collect ancient Chinese text data to generate an ancient Chinese corpus dataset. Specifically, a large amount of high-quality ancient Chinese text data can be collected to generate the ancient Chinese corpus dataset X1;
[0074] S2. Preprocess the ancient Chinese corpus dataset to generate an annotated dataset, and divide the annotated dataset into a training set and a validation set;
[0075] S3. Input the training set into the Llama 3.2 model, train the Llama 3.2 model, and expand the ancient Chinese vocabulary of the Llama 3.2 model;
[0076] S4. Take the trained Llama 3.2 model as the base model, and re-initialize the word embedding layer with the expanded Llama 3.2 model vocabulary to obtain an expanded Llama 3.2 model;
[0077] S5. Design the ancient Chinese word segmentation task and part-of-speech tagging task as sequence annotation, perform sequence annotation on the training set in step S2 to form an optimized ancient Chinese annotated corpus set;
[0078] S6. Use the optimized ancient Chinese annotated corpus set to train the expanded Llama 3.2 model obtained in step S4, and introduce a loss function to adjust the model to obtain an adjusted Llama 3.2 model;
[0079] S7. Use the validation set in step S2 to evaluate the adjusted Llama 3.2 model, and introduce a loss function to adjust the model again. Determine whether the difference between the predicted value of the model after the second adjustment and the actual value in the validation set meets the preset standard. If not, perform optimization;
[0080] S8. Use an adaptive optimization algorithm to optimize the parameters of the Llama 3.2 model whose model predicted value does not meet the preset standard, and reduce the learning rate of the model. Denote the optimized model as the separation model;
[0081] S9. Input the classical Chinese text into the segmentation model, perform classical Chinese word segmentation and part-of-speech tagging on the classical Chinese text, and output the classical Chinese text result.
[0082] Example 2
[0083] Combine Figure 1 As shown, the embodiment of the present invention provides a method for semantic understanding and segmentation of classical Chinese texts, including the following steps:
[0084] S1. Collect classical Chinese text data to generate a classical Chinese corpus dataset. Specifically, a large amount of high-quality classical Chinese text data can be collected to generate the classical Chinese corpus dataset X1.
[0085] S2. Preprocess the classical Chinese corpus dataset to generate an annotated dataset, and divide the annotated dataset into a training set and a validation set.
[0086] S3. Input the training set into the Llama 3.2 model, train the Llama 3.2 model, and expand the classical Chinese vocabulary of the Llama 3.2 model.
[0087] S4. Use the trained Llama 3.2 model as the base model, re-initialize the word embedding layer with the expanded Llama 3.2 model vocabulary to obtain the expanded Llama 3.2 model.
[0088] S5. Design the classical Chinese word segmentation task and part-of-speech tagging task as sequence annotation, perform sequence annotation on the training set in step S2 to form an optimized classical Chinese annotated corpus set.
[0089] S6. Use the optimized classical Chinese annotated corpus set to train the expanded Llama 3.2 model obtained in step S4, and introduce a loss function to adjust the model to obtain an adjusted Llama 3.2 model.
[0090] S7. Use the validation set in step S2 to evaluate the adjusted Llama 3.2 model, and introduce a loss function to adjust the model again. Determine whether the difference between the predicted value of the model after the second adjustment and the actual value in the validation set meets the preset standard. If not, perform optimization.
[0091] S8. Use an adaptive optimization algorithm to optimize the parameters of the Llama 3.2 model whose predicted value does not meet the preset standard, and reduce the learning rate of the model. Denote the optimized model as the segmentation model.
[0092] S9. Input the classical Chinese text into the segmentation model, perform classical Chinese word segmentation and part-of-speech tagging on the classical Chinese text, and output the classical Chinese text result.
[0093] On this basis, in step S2, the specific content of preprocessing the classical Chinese corpus dataset includes:
[0094] S201, Text cleaning: Remove punctuation marks, redundant characters, and incorrect content from the ancient Chinese corpus data, only retain the core text content, and at the same time unify the text format;
[0095] S202, Corpus annotation: Annotate the word segmentation boundaries and part-of-speech information for the ancient Chinese corpus dataset after text cleaning to generate an annotated dataset. Specifically, the word segmentation boundaries and part-of-speech can be manually annotated, or other AI intelligent methods can be used to annotate the word segmentation boundaries and part-of-speech information to generate an annotated dataset A1 for model training and verification.
[0096] Furthermore, in step S3, expanding the ancient Chinese vocabulary of the Llama3.2 model includes adding the vocabulary and expressions in ancient Chinese to the vocabulary of the Llama3.2 model. Specifically, adding unique words, common words, idioms, function words, unique expressions, fixed structures, etc. in ancient Chinese to the vocabulary of the Llama3.2 model to expand the ancient Chinese vocabulary of the Llama3.2 model.
[0097] Furthermore, step S5 is specifically as follows: Use the BIO (Begin-Inside-Outside) annotation method to annotate the boundaries of ancient Chinese word segmentation, obtain the ancient Chinese word segmentation, and combine the part-of-speech to form a joint part-of-speech annotation task, and perform sequence annotation on the training set to form an optimized ancient Chinese annotated corpus set;
[0098] Among them, for the BIO (Begin-Inside-Outside) annotation method, B (Begin) represents the starting position of a word, I (Inside) represents the middle part of a word, O (Outside) represents the non-word part, and at the same time additional part-of-speech information is attached. For example, VB represents a verb, NN represents a noun, AUX represents a particle, ADJ represents an adjective, etc. In addition, "B-NN" represents the starting position of a noun, "I-VB" represents the middle part of a verb, etc.
[0099] Example 3
[0100] Combined with Figure 1 As shown, the embodiment of the present invention provides an ancient Chinese text semantic understanding and segmentation method, including the following steps:
[0101] S1, Collect ancient Chinese text data to generate an ancient Chinese corpus dataset. Specifically, a large amount of high-quality ancient Chinese text data can be collected to generate an ancient Chinese corpus dataset X1;
[0102] S2, Preprocess the ancient Chinese corpus dataset to generate an annotated dataset, and divide the annotated dataset into a training set and a validation set;
[0103] S3. Input the training set into the Llama 3.2 model, train the Llama 3.2 model, and expand the classical Chinese vocabulary of the Llama 3.2 model;
[0104] S4. Use the trained Llama 3.2 model as the base model, re-initialize the word embedding layer with the expanded Llama 3.2 model vocabulary to obtain the expanded Llama 3.2 model;
[0105] S5. Design the classical Chinese word segmentation task and part-of-speech tagging task as sequence labeling, perform sequence labeling on the training set in step S2 to form an optimized classical Chinese labeled corpus;
[0106] S6. Use the optimized classical Chinese labeled corpus to train the expanded Llama 3.2 model obtained in step S4, and introduce a loss function to adjust the model to obtain an adjusted Llama 3.2 model;
[0107] S7. Use the validation set in step S2 to evaluate the adjusted Llama 3.2 model, and introduce a loss function to adjust the model again. Determine whether the difference between the predicted value of the model after the second adjustment and the actual value in the validation set meets the preset standard. If not, optimize it;
[0108] S8. Use an adaptive optimization algorithm to optimize the parameters of the Llama 3.2 model whose predicted value does not meet the preset standard, and reduce the learning rate of the model. Denote the optimized model as the segmentation model;
[0109] S9. Input the classical Chinese text into the segmentation model, perform classical Chinese word segmentation and part-of-speech tagging on the classical Chinese text, and output the classical Chinese text result;
[0110] In step S2, the specific content of preprocessing the classical Chinese corpus dataset includes:
[0111] S201. Text cleaning: Remove punctuation marks, redundant characters, and incorrect content from the classical Chinese corpus data, and unify the text format at the same time;
[0112] S202. Corpus annotation: Annotate the word segmentation boundaries and parts of speech for the classical Chinese corpus dataset after text cleaning to generate an annotated dataset;
[0113] In step S3, the classical Chinese vocabulary of the expanded Llama 3.2 model includes adding the vocabulary and expressions in classical Chinese to the vocabulary of the Llama 3.2 model;
[0114] Specifically, step S5 is to use the BIO (Begin-Inside-Outside) annotation method to annotate the boundaries of ancient Chinese word segmentation, obtain the ancient Chinese word segmentation, and combine the part-of-speech to form a joint part-of-speech annotation task, and perform sequence annotation on the training set to form an optimized ancient Chinese annotation corpus;
[0115] On this basis, in step S6, during the training process, the cross-entropy loss between the model prediction and the true annotation is minimized as the adjustment objective function to improve the accuracy of the model for ancient Chinese word segmentation and part-of-speech annotation. Specifically:
[0116]
[0117] Among them, x i represents the input ancient Chinese text, y i is the boundary and part-of-speech annotation label of the corresponding ancient Chinese word segmentation, and f Llama represents the adjusted Llama3.2 model, θ is the model parameter, and L is the cross-entropy loss function used to adjust the model parameters;
[0118]
[0119] Among them, N is the number of samples in the training set, T is the sequence length of each sample, y ij is the true label at the j-th position of the i-th sample, is the probability distribution at the j-th position of the i-th sample predicted by the model.
[0120] Example 4
[0121] Combined Figure 1 As shown, the embodiment of the present invention provides a method for semantic understanding and segmentation of ancient Chinese texts, including the following steps:
[0122] S1. Collect ancient Chinese text data to generate an ancient Chinese corpus dataset. Specifically, a large amount of high-quality ancient Chinese text data can be collected to generate the ancient Chinese corpus dataset X1;
[0123] S2. Preprocess the ancient Chinese corpus dataset to generate an annotated dataset, and divide the annotated dataset into a training set and a validation set;
[0124] S3. Input the training set into the Llama3.2 model, train the Llama3.2 model, and expand the ancient Chinese vocabulary of the Llama3.2 model;
[0125] S4. Use the trained Llama3.2 model as the base model, and re-initialize the word embedding layer with the expanded Llama3.2 model vocabulary to obtain the expanded Llama3.2 model;
[0126] S5. Design the classical Chinese word segmentation task and part-of-speech tagging task as sequence labeling, perform sequence labeling on the training set in step S2 to form an optimized classical Chinese labeled corpus set;
[0127] S6. Use the optimized classical Chinese labeled corpus set to train the extended Llama 3.2 model obtained in step S4, and introduce a loss function to adjust the model to obtain an adjusted Llama 3.2 model;
[0128] S7. Use the validation set in step S2 to evaluate the adjusted Llama 3.2 model, and introduce a loss function to adjust the model again. Determine whether the difference between the predicted value of the model after the second adjustment and the actual value in the validation set meets the preset standard. If not, optimize it;
[0129] S8. Use an adaptive optimization algorithm to optimize the parameters of the Llama 3.2 model whose predicted value does not meet the preset standard, and reduce the learning rate of the model. Denote the optimized model as the segmentation model;
[0130] S9. Input the classical Chinese text into the segmentation model, perform classical Chinese word segmentation and part-of-speech tagging on the classical Chinese text, and output the classical Chinese text result;
[0131] In step S2, the specific content of preprocessing the classical Chinese corpus dataset includes:
[0132] S201. Text cleaning: Remove punctuation marks, redundant characters, and incorrect content from the classical Chinese corpus data, and unify the text format at the same time;
[0133] S202. Corpus annotation: Annotate the boundaries of word segmentation and parts of speech for the classical Chinese corpus dataset after text cleaning to generate an annotated dataset. The parts of speech can be annotated manually, usually relying on annotation specifications or analysis of the corpus and evaluation by classical Chinese experts, and count all the part-of-speech categories in the corpus;
[0134] In step S3, the classical Chinese vocabulary of the extended Llama 3.2 model includes adding the vocabulary and expressions in classical Chinese to the vocabulary of the Llama 3.2 model;
[0135] Specifically, step S5 is to use the BIO (Begin-Inside-Outside) annotation method to annotate the boundaries of classical Chinese word segmentation to obtain classical Chinese word segmentation, and combine parts of speech to form a joint part-of-speech tagging task, perform sequence labeling on the training set to form an optimized classical Chinese labeled corpus set;
[0136] On this basis, in step S7, the specific content of evaluating the adjusted Llama 3.2 model includes:
[0137] S701. Evaluation of the Precision, Recall, and F1-Score of Ancient Chinese Word Segmentation:
[0138] The formula for Precision is expressed as follows:
[0139]
[0140] Among them, P represents Precision, TP represents the number of correctly predicted ancient Chinese word segmentation boundaries (i.e., the number of predicted ancient Chinese word segmentation boundaries that are consistent with the true ancient Chinese word segmentation boundaries), and FP represents the number of boundaries predicted as ancient Chinese word segmentation boundaries but actually not;
[0141] The formula for Recall is expressed as follows:
[0142]
[0143] Among them, FN represents the number of true word segmentation boundaries that the adjusted Llama3.2 model fails to predict;
[0144] The F1-Score is the harmonic mean of Precision and Recall. The formula for the F1-Score is expressed as follows:
[0145]
[0146] Among them, F1-Score represents the F1-Score;
[0147] S702. Evaluation of the Accuracy of Part-of-Speech Tagging:
[0148] The formula for the Accuracy of Part-of-Speech Tagging is expressed as follows:
[0149]
[0150] Among them, the number of correctly predicted tags represents the number of predicted part-of-speech tags that are consistent with the true part-of-speech tags, and the total number of tags represents the total number of all part-of-speech tags predicted using the validation set;
[0151] S703. Calculate the loss function to adjust the adjusted Llama3.2 model again. The formula is expressed as follows:
[0152]
[0153] Among them, L' represents the average cross-entropy loss value, which is used to measure the difference between the model's prediction and the true label, and is the core objective of model optimization; N' is the number of samples in the validation set, T' is the sequence length of each sample, C represents the number of categories of part-of-speech tagging, that is, the total number of all possible part-of-speech categories involved in the part-of-speech tagging task, such as "noun", "verb", "adjective", etc.; y' i'j' (k) is the true label of the k-th category at the j'-th position of the i'-th sample, and
[0154] Example 5
[0155] Combined with Figure 1 as shown, the embodiment of the present invention provides a method for separating the semantic understanding of ancient Chinese texts, including the following steps:
[0156] S1. Collect ancient Chinese text data to generate an ancient Chinese corpus dataset. Specifically, a large amount of high-quality ancient Chinese text data can be collected to generate the ancient Chinese corpus dataset X1;
[0157] S2. Preprocess the ancient Chinese corpus dataset to generate an annotated dataset, and divide the annotated dataset into a training set and a validation set;
[0158] S3. Input the training set into the Llama 3.2 model, train the Llama 3.2 model, and expand the ancient Chinese vocabulary of the Llama 3.2 model;
[0159] S4. Use the trained Llama 3.2 model as the basic model, re-initialize the word embedding layer with the expanded Llama 3.2 model vocabulary to obtain an expanded Llama 3.2 model;
[0160] S5. Design the ancient Chinese word segmentation task and the part-of-speech tagging task as sequence tagging, perform sequence tagging on the training set in step S2 to form an optimized ancient Chinese annotated corpus set;
[0161] S6. Use the optimized ancient Chinese annotated corpus set to train the expanded Llama 3.2 model obtained in step S4, and introduce a loss function to adjust the model to obtain an adjusted Llama 3.2 model;
[0162] S7. Use the validation set in step S2 to evaluate the adjusted Llama 3.2 model, and introduce a loss function to adjust the model again. Determine whether the difference between the predicted value of the model after the second adjustment and the actual value in the validation set meets the preset standard. If not, optimize it;
[0163] S8. Use an adaptive optimization algorithm to optimize the parameters of the Llama 3.2 model whose predicted values do not meet the preset standards, and reduce the learning rate of the model. Denote the optimized model as the segmentation model;
[0164] Specifically, analyze the performance of the model on complex sentence patterns or boundary - ambiguous words, adjust the model parameters, and further improve the model performance in combination with an adaptive optimization algorithm (such as AdamW);
[0165] S9. Input the ancient Chinese text into the segmentation model, perform ancient Chinese word segmentation and part - of - speech tagging on the ancient Chinese text, and output the result of the ancient Chinese text;
[0166] In step S2, the content of pre - processing the ancient Chinese corpus dataset specifically includes:
[0167] S201. Text cleaning: Remove punctuation marks, redundant characters, and incorrect content from the ancient Chinese corpus data, and unify the text format at the same time;
[0168] S202. Corpus annotation: Annotate the word segmentation boundaries and parts of speech for the ancient Chinese corpus dataset after text cleaning to generate an annotated dataset;
[0169] In step S3, expanding the ancient Chinese vocabulary table of the Llama 3.2 model includes adding the vocabulary and expressions in ancient Chinese to the vocabulary table of the Llama 3.2 model;
[0170] Step S5 is specifically to use the BIO (Begin - Inside - Outside) annotation method to annotate the boundaries of ancient Chinese word segmentation, obtain the ancient Chinese word segmentation, and combine the parts of speech to form a joint part - of - speech annotation task, and perform sequence annotation on the training set to form an optimized ancient Chinese annotated corpus set;
[0171] On this basis, step S8 specifically includes:
[0172] S801. Parameter update: Use the AdamW optimizer to update the model parameters, and the formula is expressed as follows:
[0173]
[0174] Among them, η is the learning rate, θ is the model parameter, θ t+1 The parameter of the model at the (t + 1) - th iteration of the next iteration, θ t The parameter of the model at the t - th iteration of the current iteration, and λ is the weight decay coefficient;
[0175] S802. Learning rate scheduling: Use a cosine annealing learning rate scheduler to reduce the learning rate of the model, and the formula is expressed as follows:
[0176]
[0177] Among them, η t is the current learning rate, η max is the maximum learning rate, η min is the minimum learning rate, t 次 is the current iteration number, T 次 is the total number of iterations.
[0178] Example 6
[0179] Combined with Figure 1 shown in the figure, the embodiment of the present invention provides a method for semantic understanding and segmentation of ancient Chinese text, including the following steps:
[0180] S1. Collect ancient Chinese text data to generate an ancient Chinese corpus dataset. Specifically, a large amount of high-quality ancient Chinese text data can be collected to generate the ancient Chinese corpus dataset X1;
[0181] S2. Preprocess the ancient Chinese corpus dataset to generate an annotated dataset, and divide the annotated dataset into a training set and a validation set;
[0182] S3. Input the training set into the Llama 3.2 model, train the Llama 3.2 model, and expand the ancient Chinese vocabulary of the Llama 3.2 model;
[0183] S4. Use the trained Llama 3.2 model as the basic model, and re-initialize the word embedding layer with the expanded Llama 3.2 model vocabulary to obtain the expanded Llama 3.2 model;
[0184] S5. Design the ancient Chinese word segmentation task and part-of-speech tagging task as sequence annotation, perform sequence annotation on the training set in step S2 to form an optimized ancient Chinese annotated corpus set;
[0185] S6. Use the optimized ancient Chinese annotated corpus set to train the expanded Llama 3.2 model obtained in step S4 and adjust the Llama 3.2 model;
[0186] S7. Use the validation set in step S2 to evaluate and optimize the adjusted Llama 3.2 model;
[0187] S8. Model optimization: Use an adaptive optimization algorithm to optimize the parameters of the adjusted Llama 3.2 model and reduce the learning rate of the model. Denote the optimized model as the segmentation model;
[0188] S9. Input the ancient Chinese text into the segmentation model to perform ancient Chinese word segmentation and part-of-speech tagging on the ancient Chinese text;
[0189] In step S2, the specific content of preprocessing the ancient Chinese corpus dataset includes:
[0190] S201. Text cleaning: Remove punctuation marks, redundant characters, and incorrect content from the ancient Chinese corpus data, and at the same time unify the text format;
[0191] S202. Corpus annotation: Annotate the word segmentation boundaries and part-of-speech of the ancient Chinese corpus dataset after text cleaning to generate an annotated dataset;
[0192] In step S3, expanding the ancient Chinese vocabulary list of the Llama 3.2 model includes adding the vocabulary and expressions in ancient Chinese to the vocabulary list of the Llama 3.2 model;
[0193] Specifically, step S5 is to use the BIO (Begin-Inside-Outside) annotation method to annotate the boundaries of ancient Chinese word segmentation, obtain ancient Chinese word segmentation, and combine the part-of-speech to form a combined part-of-speech annotation task, and perform sequence annotation on the training set to form an optimized ancient Chinese annotated corpus set;
[0194] On this basis, step S9 specifically includes:
[0195] S901. Input the ancient Chinese text into the segmentation model, and the segmentation model performs ancient Chinese word segmentation on the ancient Chinese text to generate a word segmentation result with word boundaries;
[0196] S902. Based on the word segmentation result with word boundaries, extract the part-of-speech of each word to generate a complete ancient Chinese semantic structure including ancient Chinese word segmentation and part-of-speech annotation;
[0197] S903. Combine the ancient Chinese word segmentation and part-of-speech, and use the segmentation model to analyze the grammar structure. Specifically, according to the part-of-speech information of each word, further analyze the grammar structure, such as subject-predicate-object relationship, modification relationship, etc., to generate a complete ancient Chinese semantic structure including ancient Chinese word segmentation and part-of-speech annotation, providing support for in-depth semantic understanding;
[0198] S904. Output the ancient Chinese text information, output the word segmentation result and part-of-speech annotation result of the ancient Chinese. Specifically, output the required ancient Chinese text information, output the ancient Chinese word segmentation result and the corresponding part-of-speech annotation, and support multiple formats (such as JSON, text, etc.) for integration into downstream applications.
[0199] This segmentation model can be applied to scenarios such as ancient Chinese translation, ancient Chinese knowledge graph construction, ancient Chinese search engines, etc., to improve the automation and intelligence level of ancient Chinese data processing.
[0200] In addition to using the Llama 3.2 model, various large language models such as BERT and GPT can also be used to implement various ancient Chinese word segmentation methods.
[0201] Example 7
[0202] An embodiment of the present invention also provides an ancient Chinese text semantic understanding and segmentation system, which uses the ancient Chinese text semantic understanding and segmentation method described above, including:
[0203] A data processing module, a model training and extension module, a data optimization module, an adjustment and evaluation module, a model optimization module, and an output module;
[0204] The data processing module is used to collect ancient Chinese text data to generate an ancient Chinese corpus dataset, preprocess the ancient Chinese corpus dataset to obtain an annotated dataset, and divide the annotated dataset into a training set and a validation set;
[0205] The model training and extension module is used to train the Llama 3.2 model using the training set obtained by the data processing module, expand the ancient Chinese vocabulary of the Llama 3.2 model, and use the trained Llama 3.2 model as the basic model to re-initialize the word embedding layer using the expanded Llama 3.2 model vocabulary to obtain an extended Llama 3.2 model;
[0206] The data optimization module is used to perform sequence annotation on the training set obtained by the data processing module to form an optimized ancient Chinese annotated corpus set;
[0207] The adjustment and evaluation module is used to train the extended Llama 3.2 model obtained by the model training and extension module using the optimized ancient Chinese annotated corpus set obtained by the data optimization module, introduce a loss function to adjust the model to obtain an adjusted Llama 3.2 model, and at the same time use the validation set obtained by the data processing module to evaluate the adjusted Llama 3.2 model, introduce a loss function to adjust the model again, and determine whether the difference between the predicted value of the model after the second adjustment and the actual value in the validation set meets the preset standard;
[0208] The model optimization module is used to optimize the parameters of the Llama 3.2 model whose predicted value obtained by the adjustment and evaluation module does not meet the preset standard, and reduce the learning rate of the model to generate a segmentation model;
[0209] The output module is used to output the result of the input ancient Chinese text, use the segmentation model obtained by the model optimization module to perform ancient Chinese word segmentation and part-of-speech tagging on the ancient Chinese text, and output the ancient Chinese text result.
[0210] The above is only an embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent replacements, and improvements made within the scope of the application of the present invention shall be included within the protection scope of the present invention.
Claims
1. A semantic understanding and segmentation method for ancient Chinese texts, characterized in that, It includes the following steps: S1. Collect ancient Chinese text data and generate an ancient Chinese corpus dataset; S2. Preprocess the ancient Chinese corpus dataset to generate an annotated dataset, and divide the annotated dataset into a training set and a validation set; S3. Input the training set into the Llama 3.2 model, train the Llama 3.2 model, and expand the ancient Chinese vocabulary of the Llama 3.2 model; S4. Take the trained Llama 3.2 model as the basic model, re-initialize the word embedding layer using the expanded vocabulary of the Llama 3.2 model to obtain an expanded Llama 3.2 model; S5. Design the ancient Chinese word segmentation task and part-of-speech tagging task as sequence annotation, perform sequence annotation on the training set in step S2 to form an optimized ancient Chinese annotated corpus set; S6. Use the optimized ancient Chinese annotated corpus set to train the expanded Llama 3.2 model obtained in step S4, and introduce a loss function to adjust the model to obtain an adjusted Llama 3.2 model; S7. Use the validation set in step S2 to evaluate the adjusted Llama 3.2 model, introduce a loss function to adjust the model again, and determine whether the difference between the predicted value of the model after the second adjustment and the actual value in the validation set meets the preset standard. If not, optimize it; S8. Use an adaptive optimization algorithm to optimize the parameters of the Llama 3.2 model whose predicted value does not meet the preset standard, and reduce the learning rate of the model. Denote the optimized model as the segmentation model; S9. Input the ancient Chinese text into the segmentation model, perform ancient Chinese word segmentation and part-of-speech tagging on the ancient Chinese text, and output the result of the ancient Chinese text.
2. The method for separating the semantic understanding of classical Chinese texts according to claim 1, characterized in that In step S2, the specific content of preprocessing the ancient Chinese corpus dataset includes: S201. Text cleaning: Remove punctuation marks, redundant characters, and incorrect content from the ancient Chinese corpus data, and unify the text format; S202. Corpus annotation: Annotate the ancient Chinese corpus dataset after text cleaning with the boundaries of ancient Chinese word segmentation and parts of speech to generate an annotated dataset.
3. The method for separating the semantic understanding of classical Chinese texts according to claim 2, wherein, In step S3, expanding the ancient Chinese vocabulary of the Llama 3.2 model includes adding the vocabulary and expressions in ancient Chinese to the vocabulary of the Llama 3.2 model.
4. The method for separating the semantic understanding of classical Chinese texts according to claim 3, characterized in that Step S5 is specifically to use the BIO annotation method to annotate the boundaries of ancient Chinese word segmentation to obtain ancient Chinese word segmentation, and combine parts of speech to form a joint part-of-speech tagging task, perform sequence annotation on the training set, and form an optimized ancient Chinese annotated corpus set.
5. The method for separating the semantic understanding of ancient Chinese text according to claim 4, characterized in that In step S6, during the training process, the cross-entropy loss between the model prediction and the true annotation is used as the adjustment objective function, specifically: Among them, x i represents the input classical Chinese text, y i is the boundary and part-of-speech tagging label of the corresponding classical Chinese word segmentation, f Llama represents the adjusted Llama 3.2 model, θ is the model parameter, and L is the cross-entropy loss function; where N is the number of samples in the training set, T is the sequence length of each sample, and y ij is the true label at the j-th position of the i-th sample, is the probability distribution at the j-th position of the i-th sample predicted by the model.
6. The method for separating the semantic understanding of classical Chinese texts according to claim 4, characterized in that, In step S7, the specific content of evaluating the adjusted Llama 3.2 model includes: S701. Evaluation of the precision, recall rate, and F1 score of ancient Chinese word segmentation: The formula for precision is expressed as follows: Where P represents precision, TP represents the number of correctly predicted ancient Chinese word segmentation boundaries, and FP represents the number of boundaries predicted as ancient Chinese word segmentation boundaries but actually not; The formula for recall rate is expressed as follows: Among them, FN represents the number of real ancient Chinese word segmentation boundaries that the adjusted Llama3.2 model fails to predict; The formula for the F1 score is expressed as follows: Among them, F1-Score represents the F1 score; S702. Evaluation of part-of-speech tagging accuracy: The formula for part-of-speech tagging accuracy is expressed as follows: Among them, Accuracy represents the part-of-speech tagging accuracy, the number of correctly predicted tags represents the number of part-of-speech tags predicted to be consistent with the real part-of-speech tags, and the total number of tags represents the total number of all part-of-speech tags predicted using the validation set; S703. Calculate the loss function to adjust the Llama3.2 model again, and the formula is expressed as follows: where, L' represents the average cross-entropy loss value, N' is the number of samples in the validation set, T' is the sequence length of each sample, C represents the number of part-of-speech categories, and y' i'j' (k) is the true label of the k-th category at the j'-th position of the i'-th sample, is the model prediction probability of the k-th category at the j'-th position of the i'-th sample.
7. The semantic understanding and segmentation method for classical Chinese text according to claim 4, characterized in that Step S8 specifically includes: S801. Parameter optimization: Update the model parameters using the AdamW optimizer, and the formula is expressed as follows: where η is the learning rate, θ is the model parameter, and θ t+1 is the parameter of the (t + 1)-th next iteration of the model, and θ t is the parameter of the t-th current iteration of the model, and λ is the weight decay coefficient; S802. Learning rate scheduling: Use the cosine annealing learning rate scheduler to reduce the learning rate of the model, and the formula is expressed as follows: Among them, η t is the current learning rate, η max is the maximum learning rate, η min is the minimum learning rate, t 次 is the current iteration number, T 次 is the total number of iterations.
8. The method for separating the semantic understanding of ancient Chinese texts according to claim 4, characterized in that, Step S9 specifically includes: S901. Input the ancient Chinese text into the segmentation model, and the segmentation model performs ancient Chinese word segmentation on the ancient Chinese text to generate an ancient Chinese word segmentation result with word boundaries; S902. Based on the ancient Chinese word segmentation result with word boundaries, extract the part of speech of each word to generate a complete ancient Chinese semantic structure including ancient Chinese word segmentation and part-of-speech tagging; S903. Combine the ancient Chinese word segmentation and part of speech, and use the segmentation model to analyze the grammatical structure; S904. Output the ancient Chinese text information, and output the ancient Chinese word segmentation result and part-of-speech tagging result.
9. An ancient text semantic understanding and segmentation system, characterized in that, Using the ancient Chinese text semantic understanding segmentation method described in any one of claims 1-8, including: A data processing module, a model training extension module, a data optimization module, an adjustment evaluation module, a model optimization module, and an output module; The data processing module is used to collect ancient Chinese text data to generate an ancient Chinese corpus dataset, preprocess the ancient Chinese corpus dataset to obtain an annotated dataset, and divide the annotated dataset into a training set and a validation set; The model training extension module is used to train the Llama3.2 model using the training set obtained by the data processing module, expand the ancient Chinese vocabulary of the Llama3.2 model, and use the trained Llama3.2 model as the basic model to re-initialize the word embedding layer using the expanded Llama3.2 model vocabulary to obtain an extended Llama3.2 model; The data optimization module is used to perform sequence annotation on the training set obtained by the data processing module to form an optimized ancient Chinese annotated corpus dataset; The adjustment evaluation module is used to use the optimized ancient Chinese annotated corpus dataset obtained by the data optimization module to train the extended Llama3.2 model obtained by the model training extension module, introduce a loss function to adjust the model to obtain an adjusted Llama3.2 model, and at the same time use the validation set obtained by the data processing module to evaluate the adjusted Llama3.2 model, and introduce a loss function to adjust the model again to determine whether the difference between the predicted value of the model after the second adjustment and the actual value in the validation set meets the preset standard; The model optimization module is used to optimize the parameters of the Llama 3.2 model whose model prediction values obtained by the adjustment evaluation module do not meet the preset standards, and reduce the learning rate of the model to generate a segmented model; The output module is used to output the results of the input classical Chinese text. Using the segmented model obtained by the model optimization module, it performs classical Chinese word segmentation and part-of-speech tagging on the classical Chinese text and outputs the results of the classical Chinese text.
Citation Information
Patent Citations
Ancient book classification method and system based on machine learning
CN119917660A