Method and system for optimizing ancient Chinese text punctuation labeling

Abstract

The invention discloses a method and a system for optimizing ancient Chinese text punctuation labeling. The method comprises the following steps: training by adopting an ancient Chinese training set after data preprocessing to obtain a plurality of ancient Chinese text labeling models; carrying out punctuation labeling on the to-be-labeled text by adopting an ancient Chinese text labeling model to obtain a plurality of first character strings; analyzing the first character string into a second character string, and configuring a plurality of punctuation mark bits for each character in the second character string; voting four punctuation mark bits of each character in the second character string according to all the first character strings by adopting a majority voting method to obtain a first voting result; and matching and splicing the second character string and the first voting result to obtain an ancient Chinese text labeling result of the to-be-labeled text. According to the method, the text labeling results of various models can be effectively fused, the final labeling decision is optimized, the accuracy and robustness of the labeling results are improved, and the method can be widely applied to the technical field of computers.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method and system for optimizing punctuation marking of ancient Chinese text. Background Art

[0002] The lack of punctuation and sentence breaks in ancient Chinese is a long-standing cultural habit. However, the text form without punctuation has become an obstacle for modern people to learn and use ancient Chinese. Manual sentence breakup needs to be based on clear semantics, grammar, rhyme, and marker words, which is time-consuming and labor-intensive. At present, a few scholars have used natural language processing technology to explore the automatic sentence breakup and punctuation of ancient Chinese. They are mainly rule-based methods, methods based on traditional probability statistics machine learning, and methods based on neural networks. After the widespread application of large models, a variety of ancient Chinese sentence breakup schemes based on large models have also emerged.

[0003] However, the hallucination problem of large models has always been a pain point for the ancient Chinese segmentation solution of large models. When the text is too long, the iterative prediction method of the large model will inevitably "talk nonsense". Not outputting according to the original classical Chinese, randomly adding, deleting and modifying words, and unexpected output interruption are common problems encountered by large models when annotating classical Chinese texts.

[0004] The traditional neural network method directly uses Bert's Chinese vector representation or retrains the Bert Chinese vector representation based on ancient Chinese, which is also a popular punctuation method. Although it can accurately output words, it is not accurate enough and has poor generalization. In addition, there are many traditional Chinese characters in ancient Chinese, and the token embedding of the traditional model generally cannot reach 90% coverage, resulting in most of the ancient Chinese being converted into unk characters, lacking semantic information. Summary of the invention

[0005] The main purpose of the embodiments of the present application is to propose a method and system for optimizing punctuation marking of ancient Chinese text, aiming to achieve a punctuation marking method with high accuracy and good robustness.

[0006] To achieve the above purpose, one aspect of the embodiments of the present application proposes a method for optimizing punctuation marking of ancient Chinese text, the method comprising:

[0007] Using the ancient Chinese training set after data preprocessing to train several ancient Chinese text annotation models; wherein each of the ancient Chinese text annotation models is constructed based on one of a large language model, a traditional model, and a neural network model;

[0008] Using the ancient Chinese text annotation model, annotating the text to be annotated with punctuation marks to obtain a plurality of first character strings;

[0009] Parsing the first string into a second string, and configuring a number of punctuation positions for each character in the second string;

[0010] Using a majority voting method, voting on the four punctuation mark positions of each character in the second character string according to all the first character strings, to obtain a first voting result;

[0011] The second character string is matched and concatenated with the first voting result to obtain an ancient Chinese text annotation result of the text to be annotated.

[0012] In some embodiments, when the ancient Chinese text annotation model is constructed based on a large language model, in the step of using the ancient Chinese text annotation model to punctuate the text to be annotated to obtain a plurality of first character strings, a character-level marker is used to perform sentence segmentation processing on the ancient Chinese training set.

[0013] In some embodiments, when the ancient Chinese text annotation model is constructed based on a large language model, in the step of using the ancient Chinese text annotation model to perform punctuation annotation on the text to be annotated to obtain a plurality of first character strings, an improved greedy search algorithm is used to search the text to be annotated, specifically including the following steps:

[0014] Constructing an initial main path according to the input text to be annotated, and performing character prediction according to the initial main path;

[0015] When the maximum probability of all results of the current predicted word in the character prediction process of the initial main path is less than a preset first probability, three first branch paths that meet the compliance judgment conditions are newly created on the initial main path;

[0016] Continue to predict along each of the first branch paths, and when the maximum probability of all results of the current prediction sub-path in the prediction process of the first branch path is less than the first probability, create three new second branch paths that meet the compliance judgment conditions on the first branch path;

[0017] The second branch path is used as the first branch path, and prediction is repeatedly performed along each of the first branch paths. When the maximum probability of all results of the current prediction sub-path in the prediction process of the first branch path is less than the first probability, three second branch paths that meet the compliance judgment conditions are newly created on the first branch path to extend the initial main path until the depth of all branch paths on the initial main path is equal to the preset depth threshold, thereby obtaining a plurality of generated branch paths;

[0018] The probabilities of all predicted words in the generated branch paths are added together to obtain a path with the highest probability as the final path, thereby completing the search for the text to be annotated.

[0019] In some embodiments, the method further includes a step of performing compliance judgment on the current predicted word, specifically including the following steps:

[0020] When the current predicted word meets the first condition, the current predicted word is determined as the selected character; wherein the first condition is that the second text is a proper subset of the text to be annotated; the second text is a text formed by connecting the first text and the current predicted character; the first text is a text formed by connecting the selected characters on the current branch path;

[0021] When the current predicted word meets the second condition, it is determined that the prediction of the current branch path ends, and the probability of the current predicted word is configured as the lowest probability; wherein the second condition is that the current predicted word is an end mark, and the first text is equal to the text to be marked;

[0022] When the currently predicted character does not meet the first condition and the second condition, select the next character with the highest probability as the currently predicted character, and repeat the step of determining the currently predicted character as the selected character when the currently predicted character meets the first condition, until the number of repeated executions reaches a preset number threshold, the currently predicted character still does not meet the first condition and the second condition, determine the character corresponding to the position of the currently predicted character from the text to be annotated as the selected character, and configure the probability of the selected character to be medium probability.

[0023] In some embodiments, the method of using a majority voting method to vote on the four punctuation marks of each character according to all of the first character strings to obtain a first voting result includes the following steps:

[0024] Determine one of the ancient Chinese text annotation models as a benchmark model;

[0025] Configure preset voting thresholds;

[0026] For the four punctuation positions of each of the characters, voting is performed on the punctuation positions according to all current characters of the first character string; when the consistency of the voting is lower than the voting threshold, the annotation result of the benchmark model is adopted as the character punctuation voting result.

[0027] When the voting for the punctuation marks of all the characters is completed, the first voting result is obtained.

[0028] In some embodiments, the method further comprises the step of preprocessing the ancient Chinese training set data, specifically comprising at least one of the following steps:

[0029] According to the characteristics of paired punctuation, regular expressions are used to clean the ancient Chinese dataset;

[0030] Arranging the ancient Chinese training set into paragraphs;

[0031] Eliminate paragraphs in which four or more punctuation marks appear consecutively in the ancient Chinese training set;

[0032] Eliminate the contents in brackets and parentheses and the punctuation marks in the contents in the ancient Chinese training set;

[0033] The traditional Chinese characters in the ancient Chinese training set are converted into simplified Chinese characters.

[0034] In some embodiments, the paragraph arrangement of the ancient Chinese training set includes the following steps:

[0035] Setting the first text in the ancient Chinese training set as a length mark; wherein the first text is a text having fewer characters than the first character number and having no punctuation marks;

[0036] Merging the second text between two adjacent length marks to obtain a third text;

[0037] The text length of the third text is adjusted to within a preset character number range to complete paragraph arrangement.

[0038] To achieve the above purpose, another aspect of the embodiment of the present application provides a system for optimizing punctuation marking of ancient Chinese text, the system comprising:

[0039] The first module is used to train a plurality of ancient Chinese text annotation models using the ancient Chinese training set after data preprocessing; wherein each of the ancient Chinese text annotation models is constructed based on one of a large language model, a traditional model, and a neural network model;

[0040] The second module is used to use the ancient Chinese text annotation model to perform punctuation annotation on the text to be annotated to obtain a plurality of first character strings;

[0041] A third module is used to parse the first string into a second string and configure a number of punctuation positions for each character in the second string;

[0042] A fourth module is configured to adopt a majority voting method to vote on the four punctuation marks of each character in the second character string according to all the first character strings to obtain a first voting result;

[0043] The fifth module is used to match and concatenate the second character string with the first voting result to obtain the ancient Chinese text annotation result of the text to be annotated.

[0044] To achieve the above objective, another aspect of an embodiment of the present application provides an electronic device, the electronic device comprising a memory and a processor, the memory storing a computer program, and the processor implementing the above-mentioned method when executing the computer program.

[0045] To achieve the above-mentioned purpose, another aspect of an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described above is implemented.

[0046] The embodiments of the present application include at least the following beneficial effects: The present application provides a method and system for optimizing punctuation marking of ancient Chinese text. The scheme obtains a first string of multiple ancient Chinese text annotation models, parses the first string into a second string, and configures punctuation marks for the characters in the second string. The majority voting method is used to vote for each punctuation mark in the second string based on the first string to obtain the final ancient Chinese text annotation result. The scheme can effectively integrate the annotation results of multiple models on the text, optimize the final annotation decision, and improve the accuracy and robustness of the annotation results. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] The accompanying drawings are used to provide further understanding of the technical solution of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the technical solution of the present application and do not constitute a limitation on the technical solution of the present application.

[0048] Figure 1 It is a schematic diagram of the steps of a method for optimizing punctuation marking of ancient Chinese text provided by an embodiment of the present application;

[0049] Figure 2 It is a schematic diagram of steps S210 to S250 provided in an embodiment of the present application;

[0050] Figure 3 It is a schematic diagram of steps S260 to S280 provided in an embodiment of the present application;

[0051] Figure 4 It is a schematic diagram of steps S410 to S440 provided in an embodiment of the present application;

[0052] Figure 5 It is a schematic diagram of steps S610 to S650 provided in an embodiment of the present application;

[0053] Figure 6It is a schematic diagram of steps S621 to S623 provided in an embodiment of the present application;

[0054] Figure 7 is a data preprocessing flow chart provided in an embodiment of the present application;

[0055] Figure 8 It is a flow chart of the improved greedy algorithm provided in the embodiment of the present application;

[0056] Figure 9 It is a module schematic diagram of a system for optimizing punctuation marking of ancient Chinese text provided by an embodiment of the present application;

[0057] Figure 10 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0058] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is further described in detail below in conjunction with the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of the present application. They are only examples of devices and methods consistent with some aspects of the embodiments of the present application as detailed in the attached claims.

[0059] Although the functional modules are divided in the system schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the system or the order in the flowchart. The terms "first / S100", "second / S200", etc. in the specification, claims and the above drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0060] It is understood that the terms "first", "second", etc. used in this application can be used to describe various concepts in this article, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another concept. For example, without departing from the scope of the embodiment of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the words "if" and "if" as used herein can be interpreted as "at the time of" or "when" or "in response to determination".

[0061] The terms "at least one", "multiple", "each", "any", etc. used in this application, at least one includes one, two or more, multiple includes two or more, each refers to each of the corresponding multiple, and any refers to any one of the multiple.

[0062] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0063] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0064] Before describing the embodiments of the present application in detail, some nouns and terms involved in the embodiments of the present application are first described. The nouns and terms involved in the embodiments of the present application are subject to the following explanations.

[0065] A large model or a large language model is an abbreviation of a large language model. A language model is an artificial intelligence model that is trained to understand and generate human language. Generally speaking, the "large" in a "large language model" means that the number of parameters of the model is very large. The large model referred to in the embodiments of the present application refers to a generative large parameter model dedicated to the field of natural language. The number of parameters may not be that large, but it is a generative model.

[0066] In related technologies, the hallucination problem of large models has always been a pain point for large models in ancient Chinese sentence segmentation solutions. When the text is too long, the iterative prediction method of the large model will inevitably "talk nonsense". Not outputting according to the original classical Chinese, adding, deleting and modifying words at will, and unexpected output interruption are common problems encountered by large models when annotating classical Chinese texts. However, traditional models do not have this problem when using sequence annotation for labeling. These two solutions have their own advantages and disadvantages, and there are the following problems:

[0067] 1. Neural network method: directly use Bert's Chinese vector representation or retrain the Bert Chinese vector representation based on ancient Chinese, which has become a popular punctuation method, with accurate word output and convenient backward parsing, but insufficient accuracy and poor generalization.

[0068] 2. Large model method: After the advent of the large model, a variety of ancient Chinese sentence segmentation solutions based on the large model have emerged one after another. Among them, the Xunzi series of large language models were open sourced in 2023 to build a large model foundation in the field of ancient Chinese. It has high accuracy and good generalization, but it is easy to add, delete, and modify words at will, the output is accidentally interrupted, and the backward parsing is troublesome.

[0069] 3. Traditional models and large models cannot be effectively integrated after annotation, and there is a lack of appropriate voting solutions.

[0070] 4. There are many ancient Chinese characters, and the traditional model token embedding generally cannot reach 90% coverage, resulting in most ancient Chinese being converted into unk characters, lacking semantic information.

[0071] In view of this, a method and system for optimizing punctuation marking of ancient Chinese text is provided in an embodiment of the present application. The scheme performs sentence segmentation on a large model through character-level token encoding, and uses a greedy search algorithm process and its supporting solution for the limited domain of ancient Chinese.

[0072] A method for optimizing punctuation marking of ancient Chinese text provided in an embodiment of the present application relates to the field of computer technology. A method for optimizing punctuation marking of ancient Chinese text provided in an embodiment of the present application can be applied to a terminal, can also be applied to a server, and can also be software running in a terminal or a server. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, and a car terminal, etc., but is not limited to this; the server side can be configured as an independent physical server, or a server cluster or distributed system composed of multiple physical servers, and can also be configured to provide cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms and other basic cloud computing services. The server can also be a node server in a blockchain network; the software can be an application that implements a method for optimizing punctuation marking of ancient Chinese text, etc., but is not limited to the above forms.

[0073] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0074] Figure 1 is an optional flow chart of a method for optimizing punctuation marking of ancient Chinese text provided in an embodiment of the present application. Figure 1 The method may include but is not limited to steps S100 to S500.

[0075] Step S100, using the ancient Chinese training set after data preprocessing to train a number of ancient Chinese text annotation models; wherein each of the ancient Chinese text annotation models is constructed based on one of a large language model, a traditional model, and a neural network model.

[0076] Step S200: using the ancient Chinese text annotation model to annotate the text with punctuation marks to obtain a plurality of first character strings.

[0077] Step S300: Parse the first character string into a second character string, and configure a number of punctuation marks for each character in the second character string.

[0078] Step S400 , using majority voting method, voting on the four punctuation marks of each character in the second character string according to all the first character strings to obtain a first voting result.

[0079] Step S500, matching and concatenating the second character string and the first voting result to obtain an ancient Chinese text annotation result of the text to be annotated.

[0080] Steps S100 to S500 shown in the embodiment of the present application obtain a first string of multiple ancient Chinese text annotation models, parse the first string into a second string, and configure punctuation marks for the characters in the second string. A majority voting method is used to vote for each punctuation mark in the second string based on the first string to obtain the final ancient Chinese text annotation result. This method can effectively integrate the annotation results of multiple models on the text, optimize the final annotation decision, and improve the accuracy and robustness of the annotation results.

[0081] In step S200 of some embodiments, when the ancient Chinese text annotation model is constructed based on a large language model, in the step of using the ancient Chinese text annotation model to punctuate the text to be annotated to obtain a plurality of first character strings, a character-level marker is used to perform sentence segmentation processing on the ancient Chinese training set.

[0082] Using a separate character-level tokenizer to segment the sentences in the ancient Chinese training set and rewriting the tokenizer loading method of the open source large model can significantly improve the accuracy of model training.

[0083] See also Figure 2 In step S200 of some embodiments, when the ancient Chinese text annotation model is constructed based on a large language model, the step of using the ancient Chinese text annotation model to punctuate the text to be annotated to obtain a plurality of first character strings uses an improved greedy search algorithm to search the text to be annotated, specifically including the following steps S210 to S250:

[0084] Step S210, constructing an initial main path according to the input text to be annotated, and performing character prediction according to the initial main path;

[0085] Step S220, when the maximum probability of all results of the current predicted word in the character prediction process of the initial main path is less than a preset first probability, three first branch paths that meet the compliance judgment condition are newly created on the initial main path;

[0086] Step S230, continue to predict along each of the first branch paths, and when the maximum probability of all results of the current prediction sub-path in the prediction process of the first branch path is less than the first probability, create three second branch paths that meet the compliance judgment conditions on the first branch path;

[0087] Step S240, taking the second branch path as the first branch path, repeatedly performing the step of predicting along each of the first branch paths, and when the maximum probability of all results of the current prediction sub-path in the prediction process of the first branch path is less than the first probability, creating three second branch paths that meet the compliance judgment conditions on the first branch path, so as to extend the initial main path until the depth of all branch paths on the initial main path is equal to the preset depth threshold, thereby obtaining a plurality of generated branch paths;

[0088] Step S250, adding the probabilities of all predicted characters in the generated branch paths to obtain the path with the highest probability as the final path, thereby completing the search for the text to be annotated.

[0089] See also Figure 3 In some embodiments, the step S200 further includes a step of performing compliance judgment on the current prediction sub-item, specifically including the following steps S260 to S280:

[0090] Step S260, when the current predicted character meets the first condition, determining the current predicted character as the selected character; wherein the first condition is that the second text is a proper subset of the text to be annotated; the second text is a text formed by connecting the first text and the current predicted character; and the first text is a text formed by connecting the selected characters on the current branch path;

[0091] Step S270, when the current predicted word meets the second condition, determining that the prediction of the current branch path ends, and configuring the probability of the current predicted word to be the lowest probability; wherein the second condition is that the current predicted word is an end mark, and the first text is equal to the text to be annotated;

[0092] Step S280, when the current predicted character does not meet the first condition and the second condition, select the next character with the highest probability as the current predicted character, and repeat the step of determining the current predicted character as the selected character when the current predicted character meets the first condition, until the number of repeated executions reaches a preset number threshold, the current predicted character still does not meet the first condition and the second condition, determine the character corresponding to the current predicted character position from the text to be annotated as the selected character, and configure the probability of the selected character to be medium probability.

[0093] By performing a greedy search including compliance judgment through the above steps S210 to S280, the problem of inconsistency between the punctuation text generated by the existing large language model during annotation and the original punctuation-free text (text addition, deletion, character order change, and even complete paragraph disappearance) can be solved, further ensuring the accuracy of the output.

[0094] In step S300 of some embodiments, the first string is parsed into a second string, and a number of punctuation positions are configured for each character in the second string. Further, the second string is the first string text without punctuation. Further, the number of punctuation positions can be configured, for example, the number of punctuation positions appearing between any two characters is usually no more than four, and the punctuation positions can be configured to four. String parsing is performed to facilitate subsequent voting processing.

[0095] See also Figure 4 In some embodiments, step S400 includes but is not limited to the following steps S410 to S440:

[0096] Step S410, determining one of the ancient Chinese text annotation models as a benchmark model;

[0097] Step S420, configuring a preset voting threshold;

[0098] Step S430, for each of the four punctuation positions of the character, vote on the punctuation positions according to all current characters of the first string, and when the consistency of the vote is lower than the voting threshold, adopt the annotation result of the benchmark model as the character punctuation voting result.

[0099] Step S440: When the voting for the punctuation marks of all the characters is completed, a first voting result is obtained.

[0100] Through the above steps S410 to S440, combined with the string concatenation processing of step S500, the results of text annotation by multiple models can be integrated, the final annotation decision can be optimized, the accuracy and robustness of the output annotation results can be improved, and texts of inconsistent lengths can also be processed, with high versatility.

[0101] Please refer to Figure 5 In some embodiments, before step S100, a step of preprocessing the ancient Chinese training set data may be included, including but not limited to at least one of the following steps S610 to S650:

[0102] Step S610, using regular expressions to clean the ancient Chinese data set according to the characteristics of paired punctuation marks;

[0103] Step S620, arranging the ancient Chinese training set into paragraphs;

[0104] Step S630, eliminating paragraphs in the ancient Chinese training set that have four or more punctuation marks appearing consecutively;

[0105] Step S640, removing the contents in brackets and parentheses and the punctuation marks in the contents in the ancient Chinese training set;

[0106] Step S650: converting the traditional Chinese characters in the ancient Chinese training set into simplified Chinese characters.

[0107] Please refer to Figure 6 In step S620 of some embodiments, the step of arranging paragraphs includes but is not limited to the following steps S621 to S623:

[0108] Step S621, setting the first text in the ancient Chinese training set as a length mark; wherein the first text is a text with fewer characters than the first character number and no punctuation;

[0109] Step S622, merging the second text between two adjacent length marks to obtain a third text;

[0110] Step S623, adjusting the text length of the third text to within a preset character number range, and completing paragraph arrangement.

[0111] Below, in combination with the application examples and specific data of the specific ancient Chinese text punctuation annotation scenario, the scheme of an embodiment of the present application is introduced and explained in detail. It should be noted that the specific parameter values ​​involved in the embodiment introduction can be adjusted according to actual needs unless otherwise specified, and the present application does not limit the specific parameter values ​​therein.

[0112] In an embodiment of the present application, a method for optimizing punctuation marking of ancient Chinese text is provided. The method can be applied to punctuation marking of ancient Chinese text, optimize the marking strategy, and improve the accuracy, robustness and versatility of the model.

[0113] For the ancient Chinese training set, usually, the training data set of ancient Chinese punctuation annotation will have a variety of data quality problems, which are specifically reflected in: (1) incorrect punctuation annotation, such as unpaired quotation marks, reversed upper and lower quotation marks, and a series of punctuation marks; (2) chaotic segmentation between chapters, no context connection; insufficient punctuation annotation, which is dirty data; (3) when there is no appropriate punctuation annotation when quoting paragraphs, some strange punctuation is used to annotate. The present application embodiment is aimed at the data preprocessing of ancient Chinese punctuation annotation and sentence segmentation, which is conducive to improving the accuracy of model training. Specifically, please refer to Figure 7 , including at least one of the following steps 1.1 to 1.7:

[0114] 1.1. Punctuation correction: According to the characteristics of paired appearance of single and double quotation marks, book titles, etc., use regular expressions to clean the training data, which can remove mislabeled data by humans and the problem that the model training is inaccurate caused by splitting a sentence with paired single and double quotation marks or book titles into multiple sentences during sentence segmentation.

[0115] 1.2. Paragraph arrangement: Set the text with less than 20 characters and no punctuation as a length marker. Then, merge the text between two adjacent length markers and adjust the length to be between 50 and 400 characters to speed up training and improve the ability to generate long texts.

[0116] 1.3. Sparse punctuation cleaning: Delete the paragraphs in the text where there is less than 1 punctuation per 10 characters on average.

[0117] 1.4. Pair detection of book titles and quotation marks: Check whether the book titles and quotation marks in the text appear in pairs, delete the parts of the paragraphs that do not conform to the paired order, and try to retain the remaining text that meets the conditions.

[0118] 1.5. Check for extremely long punctuation segments: Delete the paragraphs in the text where there are 4 or more consecutive punctuation marks, as these usually do not conform to reading habits.

[0119] Exemplarily, for the original input: Yan Shigu's annotation: "The zhu is like a deer but larger." The "Ming Yuan" says: "The large deer is called zhu..."

[0120] When there are 4 or more consecutive punctuation marks, delete the paragraph. Therefore, after data processing, the output is: Yan Shigu's annotation: "The zhu is like a deer but larger."

[0121] 1.6. Organize some words and retain the whole paragraph: Delete the content and its punctuation within the square brackets 【】 and round brackets () in the text, but retain the other text in the paragraph.

[0122] 1.7. Simplified and traditional Chinese conversion. In the embodiments of the present application, an open-source tool OpenCC and a manually compiled database containing 1000 simplified and traditional Chinese对照映射 are used for character conversion. In other embodiments, conversion can be performed through any tool capable of performing simplified and traditional Chinese conversion. Experimental evaluations of simplified and traditional Chinese conversion were carried out for the tokenizers of three language models, namely Baichuan, ChatGLM3, and Qianwen. Referring to Table 1, the experimental results show that by converting traditional Chinese characters to simplified Chinese characters, the proportion of unrecognized unknown tokens (unk token) can be significantly reduced, with the reduction range from 60% to 99%, which is beneficial to improving the accuracy of model training.

[0123] Table 1

[0124]

[0125]

[0126] When converting between simplified and traditional Chinese characters, the embodiments of the present application can combine the subsequent greedy search algorithm for ancient Chinese input restricted domain and the filtering algorithm based on multi-dimensional punctuation parsing and voting mechanism to ensure that the text can be accurately restored to its original traditional form during the conversion between simplified and traditional Chinese characters.

[0127] Furthermore, the embodiments of the present application aim to integrate the results of multiple models for text annotation and optimize the final punctuation annotation decision through the voting mechanism between models. After analyzing a large amount of text, it is found that the number of punctuation marks appearing between any two characters usually does not exceed four, and the situation beyond this number is usually caused by misannotation. Therefore, taking four punctuation positions as an example, the embodiments of the present application map the text (i.e., the first string) after punctuation annotation by a large model, a traditional model, or a neural network model to the following structure: string1 (the original text without any punctuation marks) and label1 to label4, where label1 to label4 respectively represent the first to fourth punctuation marks after each character. If there is no punctuation mark, it is represented by an empty string.

[0128] Select a base model as a reference and use the simple majority voting method to make a decision on label1 to label4, setting the voting threshold to 60%. For any character position, if the annotation results from different models do not reach 60% consistency, the annotation result of the base model is adopted.

[0129] After completing the voting process, new label1, label2, label3, and label4 will be obtained, and they will be matched and concatenated with the punctuation-free original text string1 to generate the final prediction result. This method effectively integrates the prediction results of multiple models and improves the accuracy and robustness of the annotation results.

[0130] Exemplarily, for the input of the same ancient Chinese text, when the output results of several models are as shown in Table 2 below:

[0131] Table 2

[0132] Model Input Yan Shigu's annotation: The zhu is similar to a deer but larger. The "Ming Yuan" says: The large deer is called zhu. Model Standard Output Yan Shigu's annotation: "The zhu is similar to a deer but larger." The "Ming Yuan" says: "The large deer is called zhu." Model 1 Output Yan Shigu's annotation: "The zhu, is similar to a deer but larger." The "Ming Yuan" says: "The large deer is called zhu." Model 2 Output Yan Shigu's annotation: "The zhu is similar to a deer but larger." The "Ming Yuan" says: The large deer is called zhu. Model 3 Output Yan Shigu's annotation: "The zhu is similar to a deer but larger." The "Ming Yuan" says: "The large deer is called zhu." Model 4 Output Yan Shigu's annotation: "The zhu is similar to a deer but larger." The "Ming Yuan" says: "The large deer is called zhu."

[0133] In the output results, set four empty positions after each character for four-dimensional punctuation parsing to fill the 1st, 2nd, 3rd, and 4th punctuation marks. If there is none, they are all empty. Taking the fifth character "zhu" as an example in the above example, the four-dimensional parsing results after it are shown in Table 3:

[0134] Table 3

[0135] Model 1 , empty, empty, empty Model 2 , empty, empty, empty, empty Model 3 , empty, empty, empty, empty Model 4 , empty, empty, empty, empty

[0136] Voting is carried out according to each dimension in the four dimensions in different models. If 60% of the punctuation marks can be achieved, they will be adopted. The final output result is that all four punctuation mark positions are empty.

[0137] Therefore, when the model finally outputs, there is no punctuation mark after the character "zhu", that is:

[0138] Yan Shigu annotated: "The zhu is similar to a deer but larger." "Ming Yuan" said: "The large deer is called zhu."

[0139] The method of the embodiment of the present application can effectively analyze and process the results of multiple models (especially involving the fusion of large models and the fusion of large models and traditional sequence annotation models) while improving robustness and accuracy, integrate the final annotations of all answers, especially involving the voting of non-traditional sequence annotation models, and effectively solve the problem that texts with inconsistent lengths cannot be systematically and batch-wisely integrated as a whole. Examples of texts with inconsistent lengths are as follows: text ① and text:

[0140] ① Yan Shigu annotated: "The zhu is similar to a deer but larger." "Ming Yuan" said: "The large deer is called zhu."

[0141] ② Yan Shigu annotated: "The zhu is similar to a deer and is large. Ming Yuan" said: "The large deer is called zhu."

[0142] In some embodiments, when the tokenizer encoding mechanism of a large language model (LLM) adopts word-level segmentation, this segmentation method causes incorrect merging phenomena in ancient Chinese texts without punctuation. Specifically, texts that should be divided into different units are merged into a single vocabulary, resulting in the positions where punctuation marks should be inserted not being correctly segmented and punctuated, thus having an adverse impact on the training of the model. The embodiment of the present application adopts a separate character-level token encoding method to rewrite the tokenizer loading and processing method of the open-source large model, improving the training accuracy.

[0143] During the experiment, a large language model's original tokenizer was used to process an ancient Chinese training set containing 141,374 training data (a total of 15,669,627 characters). There were 65,810 data with incorrect token situations; the characters before and after the punctuation were merged during the original encoding stage and would definitely be ignored during the model decoding stage. The number of involved vocabulary was 17,114. During this process, 134,522 out of 3,091,919 punctuation marks in the original annotation data were lost, accounting for 0.212%. The embodiment of the present application adopts a separate character-level token encoding method for sentence segmentation, which can solve the above problems and improve the accuracy of large language model training.

[0144] Exemplarily, the word segmentation results of the original tokenizer of the large model are shown in Table 4. It can be seen that after being parsed by the original tokenizer, "Da Ming" is treated as a single token, and punctuation marks between the two characters cannot be output during the output stage, resulting in problems in model training and learning.

[0145] Table 4

[0146]

[0147] The embodiments of the present application adopt a separate character-level token encoding method and rewrite the method for loading and processing the tokenizer of the open-source large model. For the original input in the above example, through the character-level token encoding method, the punctuation marks between the two characters of "Da Ming" can be input into the model for training and learning without omission.

[0148] In some embodiments, the following problems will be encountered when using the large language model for punctuation annotation: the generated representative punctuation text is inconsistent with the original unpunctuated text to be annotated, manifested as the addition, deletion, adjustment of the character order of the text, and even the disappearance of the entire text paragraph. The potential reasons for these phenomena may include: (1) the hallucination effect of the large model due to the excessive length of the text; (2) the tokenizer of the LLM may apply special encoding to some texts, such as mapping two characters to one token ID; (3) the instability of the model output.

[0149] To this end, the embodiments of the present application propose an improved greedy search algorithm based on the input ancient text in a limited domain for inference. This algorithm ensures that the LLM strictly follows the order of the input text during the inference process and punctuates word by word, thus ensuring the accuracy of the output.

[0150] Please refer to Figure 8 , define the original unpunctuated ancient Chinese text (i.e., the text to be annotated) as input_string. The specific content of the improved greedy algorithm of the present application includes:

[0151] 1. When the maximum probability of all results of the current predicted word is less than top_p during the prediction process of main_path in the main path, then create three branch paths path1 - n that meet the compliance judgment, and each path continues to perform predictions;

[0152] 2. During the branch path prediction process, if the maximum probability of all results of the current prediction word is less than top_p, then try to open three branch paths pathn-path1, pathn-path2, and pathn-path3 that meet the compliance judgment on the branch path until the depth of all branches on the main path is equal to beamn. Add the probs of all selected words in these branch paths together, find the highest path, and use it as the final path main_path. Subtract all other branch paths and continue to predict.

[0153] 3. When predicting a word tokenx in each path, compliance judgment is required:

[0154] 3.1. Assume that tokenx is selected. If the selected word in this path is concatenated and punctuation is removed, it is a true subset of input_string. Then this character is accepted.

[0155] 3.2. If tokenx is the end marker <eos>And if it is selected, check whether all the tokens in this path, when concatenated and punctuation removed, are equal to the input_string. If so, tokenx is qualified, and the probability for this step is set to 0.1, and the prediction for this path ends.

[0156] 3.3 Otherwise, select the next tokenx with the highest probability and continue the check until this process has looped 3 times in the third step and still does not meet the conditions of step 3.1 or step 3.2. Then, only set the corresponding token in the original input_string as the predicted tokenx, set the probability of the currently selected character to 0.5, and then start predicting the next character.

[0157] Exemplarily, the greedy search output in the large model source code is as shown in Table 5 below.

[0158] Table 5

[0159]

[0160] The large language model uses a generative greedy retrieval output. When outputting the 5th character, the token input to the model is: <\s> Yan Shigu annotated that the zhu is like a deer but larger <207> Yan Shigu's annotation. Here, <207> indicates that the part after <207> is the already iteratively generated answer. The candidate set for the 5th character of the large model and the probability probs of each currently predicted character are as shown in Table 6 below:

[0161] Table 6

[0162] Current Predicted Character Probability probs yue 0.521 ： 0.437 " 0.0024 zhu 0.0014

[0163] If using the greedy search output provided by the source code, the 5th character will definitely select the "曰" with the highest probs. At this time, in step 3.1 of the compliance check of the greedy search algorithm based on the input classical Chinese restricted domain proposed by the present invention, it is found that assuming tokenx is selected, the concatenated words that are definitely selected in this path, after removing punctuation, are not a proper subset of the input_string, so this tokenx is not compliant. Then the model will perform step 3.3 to select the next tokenx with the highest probability and continue to check ":". At this time, the result of the compliance check is compliant, so the model will correctly output ":".

[0164] That is, through the greedy search algorithm based on the input classical Chinese restricted domain, the actual output result of the model of the present invention is: Yan Shigu annotated: "The zhu is like a deer but larger."

[0165] It is consistent with the standard answer, thus controlling the accurate output and preventing the large model from talking nonsense and making its own assumptions.

[0166] In summary, the embodiments of the present application have at least the following beneficial effects:

[0167] 1. The four-dimensional punctuation analysis and voting filtering algorithm effectively integrates the prediction results of multiple models, improves the accuracy and robustness of the annotation results, and can effectively perform backward processing on the results of traditional models and large models. After voting, the actual verification set has an overall improvement of micro-f1 from 0.754 to 0.821.

[0168] 2. Based on the greedy search algorithm of the limited domain of input ancient text, the iterative prediction source code of the large model is transformed to fundamentally solve the problem of illusion and "nonsense" of the large model in the field of ancient text annotation. In the actual verification process, 36.4% of the problems of the large model being unable to output the original text in a standard way were solved.

[0169] 3. Modify the large model character encoding source code to solve the problem that the position where punctuation marks should be inserted cannot be correctly segmented and punctuated.

[0170] 4. The supporting ancient Chinese data cleaning solution enables more effective data cleaning and model training.

[0171] See also Figure 9 The embodiment of the present application also provides a system for optimizing punctuation marking of ancient Chinese text, which can implement the above-mentioned method for optimizing punctuation marking of ancient Chinese text, and the system includes:

[0172] The first module 101 is used to train a plurality of ancient Chinese text annotation models using the ancient Chinese training set after data preprocessing; wherein each of the ancient Chinese text annotation models is constructed based on one of a large language model, a traditional model, and a neural network model;

[0173] The second module 102 is used to use the ancient Chinese text annotation model to perform punctuation annotation on the text to be annotated to obtain a plurality of first character strings;

[0174] The third module 103 is used to parse the first character string into a second character string, and configure a number of punctuation positions for each character in the second character string;

[0175] The fourth module 104 is configured to adopt a majority voting method to vote on the four punctuation marks of each character in the second character string according to all the first character strings to obtain a first voting result;

[0176] The fifth module 105 is used to match and concatenate the second character string with the first voting result to obtain the ancient Chinese text annotation result of the text to be annotated.

[0177] It can be understood that the contents of the above method embodiments are all applicable to the present system embodiments, the functions specifically implemented by the present system embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0178] The embodiment of the present application also provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above-mentioned method for optimizing punctuation annotation of ancient Chinese text when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0179] It can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0180] See also Figure 10 , Figure 10 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0181] The processor 201 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0182] The memory 202 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 202 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 202, and the processor 201 calls and executes a method for optimizing punctuation marking of ancient Chinese text in the embodiment of this application;

[0183] Input / output interface 203, used to implement information input and output;

[0184] The communication interface 204 is used to realize the communication interaction between the device and other devices. The communication can be realized through a wired manner (such as USB, network cable, etc.) or a wireless manner (such as mobile network, WIFI, Bluetooth, etc.);

[0185] Bus 205 , which transmits information between various components of the device (e.g., processor 201 , memory 202 , input / output interface 203 , and communication interface 204 );

[0186] The processor 201 , the memory 202 , the input / output interface 203 and the communication interface 204 are connected to each other in communication within the device via the bus 205 .

[0187] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned method for optimizing punctuation marking of ancient Chinese text is implemented.

[0188] It can be understood that the contents of the above method embodiments are all applicable to the present storage medium embodiments, the functions specifically implemented by the present storage medium embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0189] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0190] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0191] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0192] The system embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment.

[0193] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0194] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0195] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or plural.

[0196] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the system embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0197] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0198] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0199] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes multiple instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, and other media that can store programs.

[0200] The preferred embodiments of the present application are described above with reference to the accompanying drawings, but the scope of the rights of the present application is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present application should be within the scope of the rights of the present application.< / eos>

Claims

1. A method for optimizing punctuation marking of ancient Chinese text, characterized in that: The following steps are involved: Using the ancient Chinese training set after data preprocessing to train several ancient Chinese text annotation models; wherein each of the ancient Chinese text annotation models is constructed based on one of a large language model, a traditional model, and a neural network model; Using the ancient Chinese text annotation model, annotating the text to be annotated with punctuation marks to obtain a plurality of first character strings; Parsing the first string into a second string, and configuring a number of punctuation positions for each character in the second string; Using a majority voting method, voting on the four punctuation mark positions of each character in the second character string according to all the first character strings, to obtain a first voting result; The second character string is matched and concatenated with the first voting result to obtain an ancient Chinese text annotation result of the text to be annotated.

2. The method according to claim 1, characterized in that: When the ancient Chinese text annotation model is constructed based on a large language model, in the step of using the ancient Chinese text annotation model to punctuate the text to be annotated to obtain a plurality of first character strings, a character-level marker is used to perform sentence segmentation processing on the ancient Chinese training set.

3. The method according to claim 1, characterized in that When the ancient Chinese text annotation model is constructed based on a large language model, in the step of using the ancient Chinese text annotation model to annotate the text to be annotated with punctuation to obtain a plurality of first character strings, an improved greedy search algorithm is used to search the text to be annotated, specifically comprising the following steps: Constructing an initial main path according to the input text to be annotated, and performing character prediction according to the initial main path; When the maximum probability of all results of the current predicted word in the character prediction process of the initial main path is less than a preset first probability, three first branch paths that meet the compliance judgment conditions are newly created on the initial main path; Continue to predict along each of the first branch paths, and when the maximum probability of all results of the current prediction sub-path in the prediction process of the first branch path is less than the first probability, create three new second branch paths that meet the compliance judgment conditions on the first branch path; The second branch path is used as the first branch path, and prediction is repeatedly performed along each of the first branch paths. When the maximum probability of all results of the current prediction sub-path in the prediction process of the first branch path is less than the first probability, three second branch paths that meet the compliance judgment conditions are newly created on the first branch path to extend the initial main path until the depth of all branch paths on the initial main path is equal to the preset depth threshold, thereby obtaining a plurality of generated branch paths; The probabilities of all predicted words in the generated branch paths are added together to obtain a path with the highest probability as the final path, thereby completing the search for the text to be annotated.

4. The method according to claim 3, characterized in that The method further comprises the step of performing compliance judgment on the current predicted word, specifically comprising the following steps: When the current predicted word meets the first condition, the current predicted word is determined as the selected character; wherein the first condition is that the second text is a proper subset of the text to be annotated; the second text is a text formed by connecting the first text and the current predicted character; the first text is a text formed by connecting the selected characters on the current branch path; When the current predicted word meets the second condition, it is determined that the prediction of the current branch path ends, and the probability of the current predicted word is configured as the lowest probability; wherein the second condition is that the current predicted word is an end mark, and the first text is equal to the text to be marked; When the currently predicted character does not meet the first condition and the second condition, select the next character with the highest probability as the currently predicted character, and repeat the step of determining the currently predicted character as the selected character when the currently predicted character meets the first condition, until the number of repeated executions reaches a preset number threshold, the currently predicted character still does not meet the first condition and the second condition, determine the character corresponding to the position of the currently predicted character from the text to be annotated as the selected character, and configure the probability of the selected character to be medium probability.

5. The method according to claim 1, characterized in that The method of adopting majority voting to vote on the four punctuation marks of each character according to all the first character strings to obtain a first voting result includes the following steps: Determine one of the ancient Chinese text annotation models as a benchmark model; Configure preset voting thresholds; For each of the four punctuation positions of the character, voting is performed on the punctuation positions according to all current characters of the first character string, and when the consistency of the voting is lower than the voting threshold, the annotation result of the benchmark model is adopted as the character punctuation voting result; When the voting for the punctuation marks of all the characters is completed, the first voting result is obtained.

6. The method according to claim 1, characterized in that The method further comprises the step of preprocessing the ancient Chinese training set data, specifically comprising at least one of the following steps: According to the characteristics of paired punctuation, regular expressions are used to clean the ancient Chinese dataset; Arranging the ancient Chinese training set into paragraphs; Eliminate paragraphs in which four or more punctuation marks appear consecutively in the ancient Chinese training set; Eliminate the contents in brackets and parentheses and the punctuation marks in the contents in the ancient Chinese training set; The traditional Chinese characters in the ancient Chinese training set are converted into simplified Chinese characters.

7. The method according to claim 6, characterized in that The paragraph arrangement of the ancient Chinese training set comprises the following steps: Setting the first text in the ancient Chinese training set as a length mark; wherein the first text is a text having fewer characters than the first character number and having no punctuation marks; Merging the second text between two adjacent length marks to obtain a third text; The text length of the third text is adjusted to within a preset character number range to complete paragraph arrangement.

8. A system for optimizing punctuation marking of ancient Chinese texts, characterized in that: include: The first module is used to train a plurality of ancient Chinese text annotation models using the ancient Chinese training set after data preprocessing; wherein each of the ancient Chinese text annotation models is constructed based on one of a large language model, a traditional model, and a neural network model; The second module is used to use the ancient Chinese text annotation model to perform punctuation annotation on the text to be annotated to obtain a plurality of first character strings; A third module is used to parse the first string into a second string and configure a number of punctuation positions for each character in the second string; A fourth module is configured to adopt a majority voting method to vote on the four punctuation marks of each character in the second character string according to all the first character strings to obtain a first voting result; The fifth module is used to match and concatenate the second character string with the first voting result to obtain the ancient Chinese text annotation result of the text to be annotated.

9. An electronic device, characterized in that: including a processor and a memory; The memory is used to store programs; The processor executes the program to implement the method according to any one of claims 1 to 7.

10. A computer storage medium storing a program executable by a processor, characterized in that: The program executable by the processor is used to implement the method according to any one of claims 1 to 7 when executed by the processor.

Images