Ancient text semantic understanding separation method and system

Abstract

The invention belongs to the technical field of text processing, and provides an ancient text semantic understanding separation method and system. Comprising the following steps: S1, collecting and generating an ancient language corpus data set; s2, performing data preprocessing, and dividing the data into a training set and a verification set; s3, inputting the training set into a Llama3.2 model for training, and expanding a vocabulary of the model; s4, re-initializing a word embedding layer by utilizing the expanded model vocabulary to obtain an expanded Llama3.2 model; s5, generating an optimized ancient language tagging corpus set; s6, expanding the Llama3.2 model to carry out training and adjustment; s7, evaluating and re-adjusting the model by using the verification set; s8, optimizing parameters of the model by using an adaptive optimization algorithm to generate a separation model; and S9, inputting the ancient Chinese text into the separation model, and performing ancient Chinese word segmentation and part-of-speech tagging on the ancient Chinese text. According to the method, the accuracy of ancient text word segmentation can be improved, accurate part-of-speech tagging is achieved, the method is suitable for complex ancient text grammatical structures, and multi-scene application extension is supported.

Description

Technical Field

[0001] The present invention belongs to the technical field of text processing, and in particular relates to a method and system for semantic understanding and separation of ancient Chinese texts. Background Art

[0002] With the rapid development of artificial intelligence technology, natural language processing (NLP) models based on deep learning have made significant progress in text semantic understanding. However, most NLP models (such as BERT, GPT series, Llama) are mainly trained for modern languages ​​and lack the ability to adapt to ancient texts. Ancient Chinese has the characteristics of concise grammar, variable word order, and complex parts of speech. Traditional semantic understanding technology has significant limitations when processing ancient Chinese. Therefore, new technical requirements are proposed for the semantic understanding of ancient texts.

[0003] At present, mainstream semantic understanding technologies (such as BERT, GPT and other pre-trained models) perform well in modern text processing. They can capture semantic relationships through context encoding, and can achieve word segmentation, part-of-speech tagging and semantic analysis. However, the training corpus of these models is mainly based on modern Chinese and English, and lacks adaptability to the characteristics of ancient Chinese. Traditional Chinese word segmentation tools (such as Jieba) have limited performance when processing ancient Chinese, mainly due to the fuzzy lexical boundaries, flexible segmentation rules, and complexity of parts of speech and syntax. In addition, the feature extraction and grammatical parsing methods of modern texts are usually based on fixed syntactic structures, and the large number of special grammatical structures such as elliptical sentences and inverted sentences in ancient Chinese make it difficult for traditional models to adapt.

[0004] However, existing technologies still have significant deficiencies in processing ancient Chinese. Pre-trained models (such as BERT and GPT) lack an understanding of the unique vocabulary and grammar of ancient Chinese, and their adaptability after fine-tuning is limited, making it difficult to effectively deal with polysemous words and complex grammar in ancient Chinese. The accuracy of word segmentation tools in ancient Chinese is low, and it is difficult to accurately identify word boundaries and sentence features. The scarcity of ancient Chinese corpus further limits the model's learning depth of ancient Chinese characteristics, resulting in poor word segmentation and part-of-speech tagging effects. In addition, conventional models have poor generalization capabilities in different types of ancient Chinese texts, and cannot effectively handle diverse ancient Chinese structures and semantics, limiting their scope of application. Summary of the invention

[0005] The present invention aims to address the technical problems existing in the background technology and provides a method and system for semantic understanding and separation of ancient Chinese texts, which can improve the accuracy of ancient Chinese word segmentation, achieve precise part-of-speech tagging, be applicable to complex ancient Chinese grammatical structures, support multi-scenario application expansion, and improve the automation and intelligence level of ancient Chinese data processing.

[0006] In order to achieve the above technical objectives, the present invention adopts the following technical solutions:

[0007] A method for semantic understanding and segmentation of ancient Chinese texts, comprising the following steps:

[0008] S1. Collect ancient Chinese text data and generate ancient Chinese corpus data set;

[0009] S2, preprocessing the ancient Chinese corpus dataset to generate an annotated dataset, and dividing the annotated dataset into a training set and a validation set;

[0010] S3, inputting the training set into the Llama3.2 model, training the Llama3.2 model, and expanding the ancient Chinese vocabulary of the Llama3.2 model;

[0011] S4, using the trained Llama3.2 model as the base model, and reinitializing the word embedding layer using the extended Llama3.2 model vocabulary to obtain the extended Llama3.2 model;

[0012] S5, designing the ancient Chinese word segmentation task and the part-of-speech tagging task as sequence tagging, performing sequence tagging on the training set in step S2, and forming an optimized ancient Chinese annotated corpus;

[0013] S6, using the optimized ancient Chinese annotated corpus to train the extended Llama3.2 model obtained in step S4, and introducing a loss function to adjust the model to obtain an adjusted Llama3.2 model;

[0014] S7, using the validation set in step S2, evaluate the adjusted Llama3.2 model, introduce the loss function to adjust the model again, and determine whether the difference between the predicted value of the adjusted model and the actual value in the validation set meets the preset standard. If not, optimize;

[0015] S8, using an adaptive optimization algorithm to optimize the parameters of the Llama3.2 model whose model prediction value does not meet the preset standard, and reducing the learning rate of the model, and recording the optimized model as a separation model;

[0016] S9. Input the ancient Chinese text into the segmentation model, perform ancient Chinese word segmentation and part-of-speech tagging on the ancient Chinese text, and output the ancient Chinese text result.

[0017] Preferably, in step S2, the content of preprocessing the ancient Chinese corpus data set specifically includes:

[0018] S201, text cleaning: remove punctuation marks, redundant characters and erroneous content in ancient text corpus data, and unify the text format;

[0019] S202, corpus annotation: annotating the ancient Chinese word segmentation boundaries and parts of speech of the ancient Chinese corpus dataset after text cleaning to generate an annotated dataset.

[0020] Preferably, in step S3, expanding the classical Chinese vocabulary of the Llama3.2 model includes adding vocabulary and expressions in classical Chinese to the vocabulary of the Llama3.2 model.

[0021] Preferably, step S5 specifically includes: using the BIO tagging method to mark the boundaries of ancient Chinese word segmentation to obtain ancient Chinese word segmentation, combining the parts of speech to form a joint part-of-speech tagging task, performing sequence tagging on the training set, and forming an optimized ancient Chinese annotated corpus.

[0022] Preferably, in step S6, during the training process, the objective function is adjusted to minimize the cross entropy loss between the model prediction and the true annotation, specifically:

[0023]

[0024] Among them, x i Indicates the input ancient text, y i is the boundary and part-of-speech tag of the corresponding ancient Chinese word segmentation, f Llama represents the adjusted Llama3.2 model, θ is the model parameter, and L is the cross entropy loss function;

[0025]

[0026] Where N is the number of samples in the training set, T is the sequence length of each sample, and y ij is the true label of the jth position of the i-th sample, is the probability distribution of the jth position of the ith sample predicted by the model.

[0027] Preferably, in step S7, the specific contents of evaluating the adjusted Llama3.2 model include:

[0028] S701, Evaluation of the precision, recall, and F1 score of ancient Chinese word segmentation:

[0029] The formula for accuracy is as follows:

[0030]

[0031] Among them, P represents the precision rate, TP represents the number of ancient Chinese word segmentation boundaries predicted correctly, and FP represents the number of ancient Chinese word segmentation boundaries predicted but not actually;

[0032] The formula for recall is as follows:

[0033]

[0034] Among them, FN represents the number of true word boundaries that the adjusted Llama3.2 model fails to predict;

[0035] The formula for the F1 score is expressed as follows:

[0036]

[0037] Among them, F1-Score represents the F1 score;

[0038] S702. Evaluation of part-of-speech tagging accuracy:

[0039] The formula for part-of-speech tagging accuracy is as follows:

[0040]

[0041] Among them, Accuracy represents the accuracy of part-of-speech tagging, the number of correctly predicted labels represents the number of predicted part-of-speech tags that are consistent with the actual part-of-speech tags, and the total number of labels represents the total number of all part-of-speech tags predicted using the validation set;

[0042] S703. Calculate the loss function to adjust the Llama3.2 model again. The formula is as follows:

[0043]

[0044] Among them, L' represents the average cross entropy loss value, N' is the number of samples in the validation set, T' is the sequence length of each sample, and C represents the number of part-of-speech categories. is the true label of the kth category at the jth position of the i'th sample, is the model predicted probability of the kth category at the jth position of the i'th sample.

[0045] Preferably, step S8 specifically includes:

[0046] S801, parameter optimization: Use AdamW optimizer to update model parameters. The formula is as follows:

[0047]

[0048] Among them, η is the learning rate, θ t+1 The parameters of the model for the next iteration t+1, θ t The parameters of the current iteration of the model, λ is the weight decay coefficient;

[0049] S802, learning rate scheduling: Use the cosine annealing learning rate scheduler to reduce the learning rate of the model. The formula is as follows:

[0050]

[0051] Among them, η t is the current learning rate, ηmax is the maximum learning rate, η min is the minimum learning rate, t 次 is the current iteration number, T 次 is the total number of iterations.

[0052] Preferably, step S9 specifically includes:

[0053] S901, inputting the classical Chinese text into the segmentation model, the segmentation model performs classical Chinese word segmentation on the classical Chinese text, and generates a word segmentation result with word boundaries;

[0054] S902, based on the word segmentation results with word boundaries, extract the part of speech of each word, and generate a complete ancient Chinese semantic structure including ancient Chinese word segmentation and part of speech tagging;

[0055] S903, Combine ancient Chinese word segmentation and part of speech, and use the separation model to analyze the grammatical structure;

[0056] S904: Output the ancient Chinese text information, and output the ancient Chinese word segmentation results and part-of-speech tagging results.

[0057] The present invention also provides a system for semantic understanding and separation of ancient Chinese texts, which uses the aforementioned method for semantic understanding and separation of ancient Chinese texts, and comprises:

[0058] Data processing module, model training extension module, data optimization module, adjustment evaluation module, model optimization module and output module;

[0059] The data processing module is used to collect ancient Chinese text data to generate an ancient Chinese corpus data set, preprocess the ancient Chinese corpus data set to obtain an annotated data set, and divide the annotated data set into a training set and a validation set;

[0060] The model training extension module is used to train the Llama3.2 model using the training set obtained by the data processing module, expand the classical Chinese vocabulary of the Llama3.2 model, and use the trained Llama3.2 model as the basic model, and use the expanded Llama3.2 model vocabulary to reinitialize the word embedding layer to obtain the extended Llama3.2 model;

[0061] The data optimization module is used to perform sequence annotation on the training set obtained by the data processing module to form an optimized ancient Chinese annotated corpus;

[0062] The adjustment and evaluation module is used to use the optimized ancient Chinese annotated corpus obtained by the data optimization module to train the extended Llama3.2 model obtained by the model training extension module, and introduce a loss function to adjust the model to obtain an adjusted Llama3.2 model. At the same time, the adjusted Llama3.2 model is evaluated using the validation set obtained by the data processing module, and the loss function is introduced to adjust the model again to determine whether the difference between the predicted value of the re-adjusted model and the actual value in the validation set meets the preset standard;

[0063] The model optimization module is used to optimize the parameters of the Llama3.2 model whose model prediction value obtained by the adjustment evaluation module does not meet the preset standard, and reduce the learning rate of the model to generate a separation model;

[0064] The output module is used to output the results of the input ancient Chinese text. It uses the segmentation model obtained by the model optimization module to perform ancient Chinese word segmentation and part-of-speech tagging on the ancient Chinese text, and outputs the ancient Chinese text results.

[0065] Compared with the prior art, the beneficial effects of the present invention are:

[0066] (1) The present invention trains, adjusts, evaluates and optimizes the Llama3.2 model, constructs a separation model suitable for ancient Chinese word segmentation and part-of-speech tagging, improves the accuracy of ancient Chinese word segmentation, achieves accurate part-of-speech tagging, is suitable for complex ancient Chinese grammatical structures, reduces the cost of ancient Chinese corpus processing, and shows significant effects in improving the semantic understanding and processing capabilities of ancient Chinese.

[0067] (2) The segmentation model constructed by the present invention supports multi-scenario application expansion and can be applied to scenarios such as ancient Chinese translation, ancient Chinese knowledge graph construction, and ancient Chinese search engines. It improves the generalization ability of the model, enhances the automation and intelligence level of ancient Chinese data processing, and provides technical support for the digitization and intelligence of ancient Chinese. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] Figure 1 A flowchart of a method for semantic understanding and segmentation of ancient Chinese text according to an embodiment of the present invention;

[0069] Figure 2 This is a technical roadmap for the ancient Chinese text semantic understanding and segmentation method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0070] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0071] Example 1

[0072] Combination Figure 1 As shown, an embodiment of the present invention provides a method for semantic understanding and segmentation of ancient Chinese text, comprising the following steps:

[0073] S1, collect ancient Chinese text data to generate an ancient Chinese corpus data set. Specifically, a large amount of high-quality ancient Chinese text data can be collected to generate an ancient Chinese corpus data set X1;

[0074] S2, preprocessing the ancient Chinese corpus dataset to generate an annotated dataset, and dividing the annotated dataset into a training set and a validation set;

[0075] S3, inputting the training set into the Llama3.2 model, training the Llama3.2 model, and expanding the ancient Chinese vocabulary of the Llama3.2 model;

[0076] S4, using the trained Llama3.2 model as the base model, and reinitializing the word embedding layer using the extended Llama3.2 model vocabulary to obtain the extended Llama3.2 model;

[0077] S5, designing the ancient Chinese word segmentation task and the part-of-speech tagging task as sequence tagging, performing sequence tagging on the training set in step S2, and forming an optimized ancient Chinese annotated corpus;

[0078] S6, using the optimized ancient Chinese annotated corpus to train the extended Llama3.2 model obtained in step S4, and introducing a loss function to adjust the model to obtain an adjusted Llama3.2 model;

[0079] S7, using the validation set in step S2, evaluate the adjusted Llama3.2 model, introduce the loss function to adjust the model again, and determine whether the difference between the predicted value of the adjusted model and the actual value in the validation set meets the preset standard. If not, optimize;

[0080] S8, using an adaptive optimization algorithm to optimize the parameters of the Llama3.2 model whose model prediction value does not meet the preset standard, and reducing the learning rate of the model, and recording the optimized model as a separation model;

[0081] S9. Input the ancient Chinese text into the segmentation model, perform ancient Chinese word segmentation and part-of-speech tagging on the ancient Chinese text, and output the ancient Chinese text result.

[0082] Example 2

[0083] Combination Figure 1 As shown, an embodiment of the present invention provides a method for semantic understanding and segmentation of ancient Chinese text, comprising the following steps:

[0084] S1, collect ancient Chinese text data to generate an ancient Chinese corpus data set. Specifically, a large amount of high-quality ancient Chinese text data can be collected to generate an ancient Chinese corpus data set X1;

[0085] S2, preprocessing the ancient Chinese corpus dataset to generate an annotated dataset, and dividing the annotated dataset into a training set and a validation set;

[0086] S3, inputting the training set into the Llama3.2 model, training the Llama3.2 model, and expanding the ancient Chinese vocabulary of the Llama3.2 model;

[0087] S4, using the trained Llama3.2 model as the base model, and reinitializing the word embedding layer using the extended Llama3.2 model vocabulary to obtain the extended Llama3.2 model;

[0088] S5, designing the ancient Chinese word segmentation task and the part-of-speech tagging task as sequence tagging, performing sequence tagging on the training set in step S2, and forming an optimized ancient Chinese annotated corpus;

[0089] S6, using the optimized ancient Chinese annotated corpus to train the extended Llama3.2 model obtained in step S4, and introducing a loss function to adjust the model to obtain an adjusted Llama3.2 model;

[0090] S7, using the validation set in step S2, evaluate the adjusted Llama3.2 model, introduce the loss function to adjust the model again, and determine whether the difference between the predicted value of the adjusted model and the actual value in the validation set meets the preset standard. If not, optimize;

[0091] S8, using an adaptive optimization algorithm to optimize the parameters of the Llama3.2 model whose model prediction value does not meet the preset standard, and reducing the learning rate of the model, and recording the optimized model as a separation model;

[0092] S9, inputting the ancient text into the separation model, performing ancient word segmentation and part-of-speech tagging on the ancient text, and outputting the ancient text result;

[0093] On this basis, in step S2, the content of preprocessing the ancient Chinese corpus dataset specifically includes:

[0094] S201, text cleaning: remove punctuation marks, redundant characters and erroneous content from ancient text corpus data, retain only the core text content, and unify the text format;

[0095] S202, corpus annotation: annotate the word segmentation boundaries and part-of-speech information of the ancient Chinese corpus dataset after text cleaning to generate an annotated dataset. Specifically, the word segmentation boundaries and part-of-speech can be annotated manually, or other AI intelligent methods can be used to annotate the word segmentation boundaries and part-of-speech information to generate an annotated dataset A1 for model training and verification.

[0096] Furthermore, in step S3, expanding the classical Chinese vocabulary of the Llama3.2 model includes adding the vocabulary and expressions in classical Chinese to the vocabulary of the Llama3.2 model, specifically, adding the unique vocabulary, common vocabulary, idioms, function words, unique expressions and fixed structures in classical Chinese to the vocabulary of the Llama3.2 model, thereby expanding the classical Chinese vocabulary of the Llama3.2 model.

[0097] Further, step S5 specifically includes: using BIO (Begin-Inside-Outside) annotation method to annotate the boundaries of ancient Chinese word segmentation, obtaining ancient Chinese word segmentation, and combining the part of speech to form a joint part of speech tagging task, performing sequence annotation on the training set, and forming an optimized ancient Chinese annotated corpus;

[0098] Among them, the BIO (Begin-Inside-Outside) annotation method, B (Begin) represents the starting position of a word, I (Inside) represents the middle part of a word, O (Outside) represents the non-word part, and additional part-of-speech information is added, such as VB represents verbs, NN represents nouns, AUX represents auxiliary words, ADJ represents adjectives, etc. In addition, "B-NN" represents the starting position of a noun, "I-VB" represents the middle part of a verb, etc.

[0099] Example 3

[0100] Combination Figure 1 As shown, an embodiment of the present invention provides a method for semantic understanding and segmentation of ancient Chinese text, comprising the following steps:

[0101] S1, collect ancient Chinese text data to generate an ancient Chinese corpus data set. Specifically, a large amount of high-quality ancient Chinese text data can be collected to generate an ancient Chinese corpus data set X1;

[0102] S2, preprocessing the ancient Chinese corpus dataset to generate an annotated dataset, and dividing the annotated dataset into a training set and a validation set;

[0103] S3, inputting the training set into the Llama3.2 model, training the Llama3.2 model, and expanding the ancient Chinese vocabulary of the Llama3.2 model;

[0104] S4, using the trained Llama3.2 model as the base model, and reinitializing the word embedding layer using the extended Llama3.2 model vocabulary to obtain the extended Llama3.2 model;

[0105] S5, designing the ancient Chinese word segmentation task and the part-of-speech tagging task as sequence tagging, performing sequence tagging on the training set in step S2, and forming an optimized ancient Chinese annotated corpus;

[0106] S6, using the optimized ancient Chinese annotated corpus to train the extended Llama3.2 model obtained in step S4, and introducing a loss function to adjust the model to obtain an adjusted Llama3.2 model;

[0107] S7, using the validation set in step S2, evaluate the adjusted Llama3.2 model, introduce the loss function to adjust the model again, and determine whether the difference between the predicted value of the adjusted model and the actual value in the validation set meets the preset standard. If not, optimize;

[0108] S8, using an adaptive optimization algorithm to optimize the parameters of the Llama3.2 model whose model prediction value does not meet the preset standard, and reducing the learning rate of the model, and recording the optimized model as a separation model;

[0109] S9, inputting the ancient text into the separation model, performing ancient word segmentation and part-of-speech tagging on the ancient text, and outputting the ancient text result;

[0110] In step S2, the content of preprocessing the ancient Chinese corpus dataset specifically includes:

[0111] S201, text cleaning: remove punctuation marks, redundant characters and erroneous content in ancient text corpus data, and unify the text format;

[0112] S202, corpus annotation: annotating word boundaries and parts of speech of the ancient Chinese corpus dataset after text cleaning to generate an annotated dataset;

[0113] In step S3, expanding the classical Chinese vocabulary of the Llama3.2 model includes adding the vocabulary and expressions in the classical Chinese to the vocabulary of the Llama3.2 model;

[0114] Step S5 specifically includes: using the BIO (Begin-Inside-Outside) annotation method to annotate the boundaries of ancient Chinese word segmentation to obtain ancient Chinese word segmentation, and combining the parts of speech to form a joint part-of-speech tagging task, performing sequence annotation on the training set to form an optimized ancient Chinese annotated corpus;

[0115] On this basis, in step S6, during the training process, the objective function is adjusted to minimize the cross entropy loss between the model prediction and the true annotation, so as to improve the accuracy of the model for ancient Chinese word segmentation and part-of-speech tagging, specifically:

[0116]

[0117] Among them, x i Indicates the input ancient text, y i is the boundary and part-of-speech tag of the corresponding ancient Chinese word segmentation, f Llama represents the adjusted Llama3.2 model, θ is the model parameter, and L is the cross entropy loss function, which is used to adjust the model parameters;

[0118]

[0119] Where N is the number of samples in the training set, T is the sequence length of each sample, and y ij is the true label of the jth position of the i-th sample, is the probability distribution of the jth position of the ith sample predicted by the model.

[0120] Example 4

[0121] Combination Figure 1 As shown, an embodiment of the present invention provides a method for semantic understanding and segmentation of ancient Chinese text, comprising the following steps:

[0122] S1, collect ancient Chinese text data to generate an ancient Chinese corpus data set. Specifically, a large amount of high-quality ancient Chinese text data can be collected to generate an ancient Chinese corpus data set X1;

[0123] S2, preprocessing the ancient Chinese corpus dataset to generate an annotated dataset, and dividing the annotated dataset into a training set and a validation set;

[0124] S3, inputting the training set into the Llama3.2 model, training the Llama3.2 model, and expanding the ancient Chinese vocabulary of the Llama3.2 model;

[0125] S4, using the trained Llama3.2 model as the base model, and reinitializing the word embedding layer using the extended Llama3.2 model vocabulary to obtain the extended Llama3.2 model;

[0126] S5, designing the ancient Chinese word segmentation task and the part-of-speech tagging task as sequence tagging, performing sequence tagging on the training set in step S2, and forming an optimized ancient Chinese annotated corpus;

[0127] S6, using the optimized ancient Chinese annotated corpus to train the extended Llama3.2 model obtained in step S4, and introducing a loss function to adjust the model to obtain an adjusted Llama3.2 model;

[0128] S7, using the validation set in step S2, evaluate the adjusted Llama3.2 model, introduce the loss function to adjust the model again, and determine whether the difference between the predicted value of the adjusted model and the actual value in the validation set meets the preset standard. If not, optimize;

[0129] S8, using an adaptive optimization algorithm to optimize the parameters of the Llama3.2 model whose model prediction value does not meet the preset standard, and reducing the learning rate of the model, and recording the optimized model as a separation model;

[0130] S9, inputting the ancient text into the separation model, performing ancient word segmentation and part-of-speech tagging on the ancient text, and outputting the ancient text result;

[0131] In step S2, the content of preprocessing the ancient Chinese corpus dataset specifically includes:

[0132] S201, text cleaning: remove punctuation marks, redundant characters and erroneous content in ancient text corpus data, and unify the text format;

[0133] S202, corpus annotation: annotate the word boundaries and parts of speech of the ancient Chinese corpus dataset after text cleaning to generate an annotated dataset. The parts of speech can be annotated manually, usually relying on annotation specifications or analysis of the corpus and evaluation by ancient Chinese experts, and all parts of speech categories in the corpus are counted;

[0134] In step S3, expanding the classical Chinese vocabulary of the Llama3.2 model includes adding the vocabulary and expressions in the classical Chinese to the vocabulary of the Llama3.2 model;

[0135] Step S5 specifically includes: using the BIO (Begin-Inside-Outside) annotation method to annotate the boundaries of ancient Chinese word segmentation to obtain ancient Chinese word segmentation, and combining the parts of speech to form a joint part-of-speech tagging task, performing sequence annotation on the training set to form an optimized ancient Chinese annotated corpus;

[0136] On this basis, in step S7, the specific contents of evaluating the adjusted Llama3.2 model include:

[0137] S701, Evaluation of the Precision, Recall, and F1-Score of Ancient Chinese Word Segmentation:

[0138] The formula for precision is as follows:

[0139]

[0140] Among them, P represents the precision rate, TP represents the number of correctly predicted ancient Chinese word segmentation boundaries (that is, the number of predicted ancient Chinese word segmentation boundaries that are consistent with the actual ancient Chinese word segmentation boundaries), and FP represents the number of predicted ancient Chinese word segmentation boundaries but actually not ancient Chinese word segmentation boundaries;

[0141] The formula for recall is as follows:

[0142]

[0143] Among them, FN represents the number of true word boundaries that the adjusted Llama3.2 model fails to predict;

[0144] The F1 score is the harmonic average of precision and recall. The formula for the F1 score (F1-Score) is as follows:

[0145]

[0146] Among them, F1-Score represents the F1 score;

[0147] S702. Evaluation of part-of-speech tagging accuracy:

[0148] The formula for part-of-speech tagging accuracy is as follows:

[0149]

[0150] Among them, the number of correctly predicted labels indicates the number of predicted part-of-speech tags that are consistent with the true part-of-speech tags, and the total number of labels indicates the total number of all part-of-speech tags predicted using the validation set;

[0151] S703, calculate the loss function to adjust the adjusted Llama3.2 model again, and the formula is as follows:

[0152]

[0153] Among them, L' represents the average cross entropy loss value, which is used to measure the difference between the model's prediction and the true label, and is the core goal of model optimization; N' is the number of samples in the validation set, T' is the sequence length of each sample, and C represents the number of part-of-speech tagging categories, that is, the total number of all possible part-of-speech categories involved in the part-of-speech tagging task, such as "noun", "verb", "adjective", etc.; y' i'j' (k) is the true label of the kth category at the jth position of the i'th sample, is the model predicted probability of the kth category at the jth position of the i'th sample.

[0154] Example 5

[0155] Combination Figure 1 As shown, an embodiment of the present invention provides a method for semantic understanding and segmentation of ancient Chinese text, comprising the following steps:

[0156] S1, collect ancient Chinese text data to generate an ancient Chinese corpus data set. Specifically, a large amount of high-quality ancient Chinese text data can be collected to generate an ancient Chinese corpus data set X1;

[0157] S2, preprocessing the ancient Chinese corpus dataset to generate an annotated dataset, and dividing the annotated dataset into a training set and a validation set;

[0158] S3, inputting the training set into the Llama3.2 model, training the Llama3.2 model, and expanding the ancient Chinese vocabulary of the Llama3.2 model;

[0159] S4, using the trained Llama3.2 model as the base model, and reinitializing the word embedding layer using the extended Llama3.2 model vocabulary to obtain the extended Llama3.2 model;

[0160] S5, designing the ancient Chinese word segmentation task and the part-of-speech tagging task as sequence tagging, performing sequence tagging on the training set in step S2, and forming an optimized ancient Chinese annotated corpus;

[0161] S6, using the optimized ancient Chinese annotated corpus to train the extended Llama3.2 model obtained in step S4, and introducing a loss function to adjust the model to obtain an adjusted Llama3.2 model;

[0162] S7, using the validation set in step S2, evaluate the adjusted Llama3.2 model, introduce the loss function to adjust the model again, and determine whether the difference between the predicted value of the adjusted model and the actual value in the validation set meets the preset standard. If not, optimize;

[0163] S8, using an adaptive optimization algorithm to optimize the parameters of the Llama3.2 model whose model prediction value does not meet the preset standard, and reducing the learning rate of the model, and recording the optimized model as a separation model;

[0164] Specifically, the performance of the model on complex sentences or boundary-fuzzy vocabulary is analyzed, model parameters are adjusted, and adaptive optimization algorithms (such as AdamW) are combined to further improve model performance;

[0165] S9, inputting the ancient text into the separation model, performing ancient word segmentation and part-of-speech tagging on the ancient text, and outputting the ancient text result;

[0166] In step S2, the content of preprocessing the ancient Chinese corpus dataset specifically includes:

[0167] S201, text cleaning: remove punctuation marks, redundant characters and erroneous content in ancient text corpus data, and unify the text format;

[0168] S202, corpus annotation: annotating word boundaries and parts of speech of the ancient Chinese corpus dataset after text cleaning to generate an annotated dataset;

[0169] In step S3, expanding the classical Chinese vocabulary of the Llama3.2 model includes adding the vocabulary and expressions in the classical Chinese to the vocabulary of the Llama3.2 model;

[0170] Step S5 specifically includes: using the BIO (Begin-Inside-Outside) annotation method to annotate the boundaries of ancient Chinese word segmentation to obtain ancient Chinese word segmentation, and combining the parts of speech to form a joint part-of-speech tagging task, performing sequence annotation on the training set to form an optimized ancient Chinese annotated corpus;

[0171] On this basis, step S8 specifically includes:

[0172] S801, parameter update: Use AdamW optimizer to update model parameters. The formula is as follows:

[0173]

[0174] Among them, η is the learning rate, θ is the model parameter, and θ t+1 The parameters of the model for the next iteration t+1, θ t The parameters of the current iteration of the model, λ is the weight decay coefficient;

[0175] S802, learning rate scheduling: Use the cosine annealing learning rate scheduler to reduce the learning rate of the model. The formula is as follows:

[0176]

[0177] Among them, η t is the current learning rate, η max is the maximum learning rate, η min is the minimum learning rate, t 次 is the current iteration number, T 次 is the total number of iterations.

[0178] Example 6

[0179] Combination Figure 1 As shown, an embodiment of the present invention provides a method for semantic understanding and segmentation of ancient Chinese text, comprising the following steps:

[0180] S1, collect ancient Chinese text data to generate an ancient Chinese corpus data set. Specifically, a large amount of high-quality ancient Chinese text data can be collected to generate an ancient Chinese corpus data set X1;

[0181] S2, preprocessing the ancient Chinese corpus dataset to generate an annotated dataset, and dividing the annotated dataset into a training set and a validation set;

[0182] S3, inputting the training set into the Llama3.2 model, training the Llama3.2 model, and expanding the ancient Chinese vocabulary of the Llama3.2 model;

[0183] S4, using the trained Llama3.2 model as the base model, and reinitializing the word embedding layer using the extended Llama3.2 model vocabulary to obtain the extended Llama3.2 model;

[0184] S5, designing the ancient Chinese word segmentation task and the part-of-speech tagging task as sequence tagging, performing sequence tagging on the training set in step S2, and forming an optimized ancient Chinese annotated corpus;

[0185] S6, using the optimized ancient Chinese annotated corpus, training the extended Llama3.2 model obtained in step S4, and adjusting the Llama3.2 model;

[0186] S7. Use the validation set in step S2 to evaluate and optimize the adjusted Llama3.2 model.

[0187] S8, model optimization: use the adaptive optimization algorithm to optimize the parameters of the adjusted Llama3.2 model, reduce the learning rate of the model, and record the optimized model as the separation model;

[0188] S9, inputting the ancient Chinese text into the segmentation model, and performing ancient Chinese word segmentation and part-of-speech tagging on the ancient Chinese text;

[0189] In step S2, the content of preprocessing the ancient Chinese corpus dataset specifically includes:

[0190] S201, text cleaning: remove punctuation marks, redundant characters and erroneous content in ancient text corpus data, and unify the text format;

[0191] S202, corpus annotation: annotating word boundaries and parts of speech of the ancient Chinese corpus dataset after text cleaning to generate an annotated dataset;

[0192] In step S3, expanding the classical Chinese vocabulary of the Llama3.2 model includes adding the vocabulary and expressions in the classical Chinese to the vocabulary of the Llama3.2 model;

[0193] Step S5 specifically includes: using the BIO (Begin-Inside-Outside) annotation method to annotate the boundaries of ancient Chinese word segmentation to obtain ancient Chinese word segmentation, and combining the parts of speech to form a joint part-of-speech tagging task, performing sequence annotation on the training set to form an optimized ancient Chinese annotated corpus;

[0194] On this basis, step S9 specifically includes:

[0195] S901, inputting the classical Chinese text into the segmentation model, the segmentation model performs classical Chinese word segmentation on the classical Chinese text, and generates a word segmentation result with word boundaries;

[0196] S902, based on the word segmentation results with word boundaries, extract the part of speech of each word, and generate a complete ancient Chinese semantic structure including ancient Chinese word segmentation and part of speech tagging;

[0197] S903, combining ancient Chinese word segmentation and part of speech, using the separation model to analyze the grammatical structure, specifically, according to the part of speech information of each word, further analyzing the grammatical structure, such as subject-verb-object relationship, modification relationship, etc., to generate a complete ancient Chinese semantic structure including ancient Chinese word segmentation and part of speech tagging, to provide support for deep semantic understanding;

[0198] S904, output ancient Chinese text information, output ancient Chinese word segmentation results and part-of-speech tagging results. Specifically, output the required ancient Chinese text information, output ancient Chinese word segmentation results and corresponding part-of-speech tagging, and support multiple formats (such as JSON, text, etc.) for integration into downstream applications.

[0199] This separation model can be applied to scenarios such as ancient Chinese translation, ancient Chinese knowledge graph construction, and ancient Chinese search engines, to improve the level of automation and intelligence in ancient Chinese data processing.

[0200] In addition to using the Llama3.2 model, you can also use BERT, GPT and other large language models to implement a variety of ancient Chinese word segmentation methods.

[0201] Example 7

[0202] The embodiment of the present invention further provides a system for semantic understanding and segmentation of ancient Chinese texts, which uses the aforementioned method for semantic understanding and segmentation of ancient Chinese texts, including:

[0203] Data processing module, model training extension module, data optimization module, adjustment evaluation module, model optimization module and output module;

[0204] The data processing module is used to collect ancient Chinese text data to generate an ancient Chinese corpus data set, preprocess the ancient Chinese corpus data set to obtain an annotated data set, and divide the annotated data set into a training set and a validation set;

[0205] The model training extension module is used to train the Llama3.2 model using the training set obtained by the data processing module, expand the classical Chinese vocabulary of the Llama3.2 model, and use the trained Llama3.2 model as the basic model, and use the expanded Llama3.2 model vocabulary to reinitialize the word embedding layer to obtain the extended Llama3.2 model;

[0206] The data optimization module is used to perform sequence annotation on the training set obtained by the data processing module to form an optimized ancient Chinese annotated corpus;

[0207] The adjustment and evaluation module is used to use the optimized ancient Chinese annotated corpus obtained by the data optimization module to train the extended Llama3.2 model obtained by the model training extension module, and introduce a loss function to adjust the model to obtain an adjusted Llama3.2 model. At the same time, the adjusted Llama3.2 model is evaluated using the validation set obtained by the data processing module, and the loss function is introduced to adjust the model again to determine whether the difference between the predicted value of the re-adjusted model and the actual value in the validation set meets the preset standard;

[0208] The model optimization module is used to optimize the parameters of the Llama3.2 model whose model prediction value obtained by the adjustment evaluation module does not meet the preset standard, and reduce the learning rate of the model to generate a separation model;

[0209] The output module is used to output the results of the input ancient Chinese text. It uses the segmentation model obtained by the model optimization module to perform ancient Chinese word segmentation and part-of-speech tagging on the ancient Chinese text, and outputs the ancient Chinese text results.

[0210] The above description is only an embodiment of the present invention and is not intended to limit the present invention. Any modification, equivalent replacement and improvement made within the application scope of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for semantic understanding and segmentation of ancient Chinese texts, characterized in that: The following steps are involved: S1. Collect ancient Chinese text data and generate ancient Chinese corpus data set; S2, preprocessing the ancient Chinese corpus dataset to generate an annotated dataset, and dividing the annotated dataset into a training set and a validation set; S3, inputting the training set into the Llama3.2 model, training the Llama3.2 model, and expanding the ancient Chinese vocabulary of the Llama3.2 model; S4, using the trained Llama3.2 model as the base model, and reinitializing the word embedding layer using the extended Llama3.2 model vocabulary to obtain the extended Llama3.2 model; S5, designing the ancient Chinese word segmentation task and the part-of-speech tagging task as sequence tagging, performing sequence tagging on the training set in step S2, and forming an optimized ancient Chinese annotated corpus; S6, using the optimized ancient Chinese annotated corpus to train the extended Llama3.2 model obtained in step S4, and introducing a loss function to adjust the model to obtain an adjusted Llama3.2 model; S7, using the validation set in step S2, evaluate the adjusted Llama3.2 model, introduce the loss function to adjust the model again, and determine whether the difference between the predicted value of the adjusted model and the actual value in the validation set meets the preset standard. If not, optimize; S8, using an adaptive optimization algorithm to optimize the parameters of the Llama3.2 model whose prediction value does not meet the preset standard, and reducing the learning rate of the model, and recording the optimized model as a separation model; S9. Input the ancient Chinese text into the segmentation model, perform ancient Chinese word segmentation and part-of-speech tagging on the ancient Chinese text, and output the ancient Chinese text result.

2. The method for semantic understanding and segmentation of ancient Chinese text according to claim 1, characterized in that: In step S2, the content of preprocessing the ancient Chinese corpus dataset specifically includes: S201, text cleaning: remove punctuation marks, redundant characters and erroneous content in ancient text corpus data, and unify the text format; S202, corpus annotation: annotating the ancient Chinese word segmentation boundaries and parts of speech of the ancient Chinese corpus dataset after text cleaning to generate an annotated dataset.

3. The method for semantic understanding and segmentation of ancient Chinese text according to claim 2 is characterized in that: In step S3, expanding the classical Chinese vocabulary of the Llama3.2 model includes adding the vocabulary and expressions in classical Chinese to the vocabulary of the Llama3.2 model.

4. The method for semantic understanding and segmentation of ancient Chinese text according to claim 3 is characterized in that: Step S5 specifically includes using the BIO tagging method to mark the boundaries of ancient Chinese word segments to obtain ancient Chinese word segments, combining the parts of speech to form a joint part-of-speech tagging task, performing sequence tagging on the training set, and forming an optimized ancient Chinese annotated corpus.

5. The method for semantic understanding and segmentation of ancient Chinese text according to claim 4 is characterized in that: In step S6, during the training process, the objective function is adjusted to minimize the cross entropy loss between the model prediction and the true annotation, specifically: Among them, x i Indicates the input ancient text, y i is the boundary and part-of-speech tag of the corresponding ancient Chinese word segmentation, f Llama represents the adjusted Llama3.2 model, θ is the model parameter, and L is the cross entropy loss function; Where N is the number of samples in the training set, T is the sequence length of each sample, and y ij is the true label of the jth position of the i-th sample, is the probability distribution of the jth position of the ith sample predicted by the model.

6. The method for semantic understanding and segmentation of ancient Chinese text according to claim 4, characterized in that: In step S7, the specific contents of evaluating the adjusted Llama3.2 model include: S701, Evaluation of the precision, recall, and F1 score of ancient Chinese word segmentation: The formula for accuracy is as follows: Among them, P represents the precision rate, TP represents the number of ancient Chinese word segmentation boundaries predicted correctly, and FP represents the number of ancient Chinese word segmentation boundaries predicted but not actually; The formula for recall is as follows: Among them, FN represents the number of true ancient Chinese word segmentation boundaries that the adjusted Llama3.2 model fails to predict; The formula for the F1 score is expressed as follows: Among them, F1-Score represents the F1 score; S702. Evaluation of part-of-speech tagging accuracy: The formula for part-of-speech tagging accuracy is as follows: Among them, Accuracy represents the accuracy of part-of-speech tagging, the number of correctly predicted labels represents the number of predicted part-of-speech tags that are consistent with the actual part-of-speech tags, and the total number of labels represents the total number of all part-of-speech tags predicted using the validation set; S703. Calculate the loss function to adjust the Llama3.2 model again. The formula is as follows: Among them, L' represents the average cross entropy loss value, N' is the number of samples in the validation set, T' is the sequence length of each sample, C represents the number of categories of part of speech, and y' i'j' (k) is the true label of the kth category at the jth position of the i'th sample, is the model predicted probability of the kth category at the j′th position of the i′th sample.

7. The method for semantic understanding and segmentation of ancient Chinese text according to claim 4, characterized in that: Step S8 specifically includes: S801, parameter optimization: Use AdamW optimizer to update model parameters. The formula is as follows: Among them, η is the learning rate, θ is the model parameter, and θ t+1 The parameters of the model for the next iteration t+1, θ t The parameters of the current iteration of the model, λ is the weight decay coefficient; S802, learning rate scheduling: Use the cosine annealing learning rate scheduler to reduce the learning rate of the model. The formula is as follows: Among them, η t is the current learning rate, η max is the maximum learning rate, η min is the minimum learning rate, t 次 is the current iteration number, T 次 is the total number of iterations.

8. The method for semantic understanding and segmentation of ancient Chinese text according to claim 4, characterized in that: Step S9 specifically includes: S901, inputting the ancient Chinese text into the segmentation model, the segmentation model performs ancient Chinese word segmentation on the ancient Chinese text, and generates an ancient Chinese word segmentation result with word boundaries; S902, based on the ancient Chinese word segmentation results with word boundaries, extract the part of speech of each word, and generate a complete ancient Chinese semantic structure including the ancient Chinese word segmentation and part of speech tagging; S903, Combine ancient Chinese word segmentation and part of speech, and use the separation model to analyze the grammatical structure; S904: Output the ancient Chinese text information, and output the ancient Chinese word segmentation results and part-of-speech tagging results.

9. A system for semantic understanding and separation of ancient Chinese texts, characterized in that: The method for semantic understanding and segmentation of ancient Chinese text as claimed in any one of claims 1 to 8 comprises: Data processing module, model training extension module, data optimization module, adjustment evaluation module, model optimization module and output module; The data processing module is used to collect ancient Chinese text data to generate an ancient Chinese corpus data set, preprocess the ancient Chinese corpus data set to obtain an annotated data set, and divide the annotated data set into a training set and a validation set; The model training extension module is used to train the Llama3.2 model using the training set obtained by the data processing module, expand the classical Chinese vocabulary of the Llama3.2 model, and use the trained Llama3.2 model as the basic model, and use the expanded Llama3.2 model vocabulary to reinitialize the word embedding layer to obtain the extended Llama3.2 model; The data optimization module is used to perform sequence annotation on the training set obtained by the data processing module to form an optimized ancient Chinese annotated corpus; The adjustment and evaluation module is used to use the optimized ancient Chinese annotated corpus obtained by the data optimization module to train the extended Llama3.2 model obtained by the model training extension module, and introduce a loss function to adjust the model to obtain an adjusted Llama3.2 model. At the same time, the adjusted Llama3.2 model is evaluated using the validation set obtained by the data processing module, and the loss function is introduced to adjust the model again to determine whether the difference between the predicted value of the re-adjusted model and the actual value in the validation set meets the preset standard; The model optimization module is used to optimize the parameters of the Llama3.2 model whose model prediction value obtained by the adjustment evaluation module does not meet the preset standard, and reduce the learning rate of the model to generate a separation model; The output module is used to output the results of the input ancient Chinese text. It uses the segmentation model obtained by the model optimization module to perform ancient Chinese word segmentation and part-of-speech tagging on the ancient Chinese text, and outputs the ancient Chinese text results.

Images