Ancient book text survey method and system based on large language model

Abstract

The invention discloses an ancient book text surveying method and system based on a large language model, and relates to the technical field of text processing, and the method comprises the steps: obtaining a to-be-surveyed ancient book text with a mask mark; performing vector conversion on the ancient book text to be investigated according to the titles, and determining a plurality of vector text sentences of each title; performing mask prediction on the vector text sentences with the mask marks based on a Chinese general word list through a pre-trained large language model, and outputting mask prediction word probability distribution of each mask mark; performing reconnaissance on the associated vector text sentences with the mask marks by adopting the probability distribution of the mask prediction words, and outputting corresponding first reconnaissance vector text sentences; and if the reconnaissance accuracy of any first reconnaissance vector text sentence is greater than or equal to a preset accuracy threshold, generating a reconnaissance ancient book text according to each first reconnaissance vector text sentence. And intelligent correction is performed on the to-be-corrected text by utilizing strong context understanding and text prediction capabilities of the large language model, so that the correction efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of text processing, and in particular to an ancient book text proofreading method and system based on a large language model. Background Art

[0002] Ancient books are treasures of human civilization. In the current wave of digital humanities research, the digitization of ancient books has become an irreversible trend. In the process of ancient books being circulated, there are often word errors due to copying, reprinting, etc., intentional or unintentional, which leads to the change of the original meaning of the ancient books. Therefore, by using different versions or other materials to collate ancient books, errors in the documents can be discovered and corrected in order to restore the original appearance of the ancient books.

[0003] The current work of ancient book collation still relies on expert experience and manually proofreads massive amounts of ancient book text data, resulting in low efficiency in ancient book collation. Summary of the invention

[0004] The present invention provides an ancient book text collation method and system based on a large language model, which solves the technical problem that the current ancient book collation work still relies on expert experience and manually collates massive ancient book text data, resulting in low efficiency of ancient book collation.

[0005] The first aspect of the present invention provides a method for proofreading ancient texts based on a large language model, comprising:

[0006] Obtain the ancient book text to be proofread with mask marks;

[0007] Performing vector conversion on the ancient book text to be collated according to the title, and determining a plurality of vector text sentences of each title;

[0008] Performing mask prediction on vector text sentences with mask tags based on a Chinese general vocabulary using a pre-trained large language model, and outputting a probability distribution of mask prediction words for each mask tag;

[0009] Using each of the mask prediction word probability distributions to collate the associated vector text sentence with the mask mark, and outputting a corresponding first collated vector text sentence;

[0010] If the collation accuracy of any of the first collated vector text sentences is greater than or equal to a preset accuracy threshold, a collated ancient book text is generated according to each of the first collated vector text sentences.

[0011] Optionally, performing vector conversion on the text of the ancient book to be collated and proofread according to the title to determine a plurality of vector text sentences of each title includes:

[0012] Standardizing the punctuation marks of the ancient book text to be proofread and outputting the standardized ancient book text to be proofread;

[0013] Merging the text contents of different titles in the standardized ancient book text to be proofread with line breaks as boundaries to generate a converted ancient book text to be proofread;

[0014] After traversing the converted ancient book text to be collated and proofread in character units and removing prohibited characters, vector conversion is performed to determine a plurality of vector text sentences of each of the titles.

[0015] Optionally, performing mask prediction on the vector text sentences with mask marks based on the Chinese general vocabulary by using the pre-trained large language model, and outputting the probability distribution of mask prediction words of each mask mark, comprises:

[0016] When the vector text sentence with the mask mark is the first vector text sentence of the title to which it belongs, the vector text sentence with the mask mark is concatenated with the associated lower vector text sentence to obtain the vector text sentence to be input;

[0017] When the vector text sentence with the mask mark is the last vector text sentence of the title to which it belongs, the vector text sentence with the mask mark is concatenated with the associated upper vector text sentence to generate the vector text sentence to be input;

[0018] When the vector text sentence with the mask mark is an intermediate vector text sentence of the title to which it belongs, the vector text sentence with the mask mark is concatenated with the associated upper vector text sentence and lower vector text sentence to determine the vector text sentence to be input;

[0019] Each of the input vector text sentences is sequentially input into the pre-trained large language model, and the probability distribution of the masked predicted words of each masked tag with respect to the Chinese general vocabulary is correspondingly output.

[0020] Optionally, the using the probability distribution of each masked predicted word to collate the associated vector text sentence with the mask mark, and outputting the corresponding first collated vector text sentence, comprises:

[0021] Based on the probability distribution of each of the mask prediction words, a plurality of mask prediction words with a preset number threshold are selected in descending order of probability, corresponding to the verification candidate word sets constituting each of the mask marks;

[0022] If the masked word corresponding to the mask mark belongs to the associated verification candidate word set, the masked word is filled into the position of the mask mark, and the corresponding first proofread vector text sentence is output;

[0023] If the masked word corresponding to the mask mark does not belong to the associated verification candidate word set, the mask prediction word with the highest probability is selected from the verification candidate word set to fill the position of the mask mark, and the corresponding first proofread vector text sentence is output.

[0024] Optionally, it also includes:

[0025] If the collation accuracy of any of the first collated vector text sentences is less than a preset accuracy threshold, selecting a first collated standard ancient book text according to the attribute information of the ancient book text to be collated to perform incremental learning on the pre-trained large language model, and outputting an optimized large language model;

[0026] Using the optimized large language model to perform mask prediction on the vector text sentence with the mask mark based on the Chinese general vocabulary, until a second proofread vector text sentence is output;

[0027] If the proofreading accuracy of any of the second proofread vector text sentences is greater than or equal to a preset accuracy threshold, a proofread ancient book text is generated according to each of the second proofread vector text sentences.

[0028] Optionally, it also includes:

[0029] If the proofreading accuracy of any of the second proofread vector text sentences is less than a preset accuracy threshold, then according to the vector text sentence with the mask mark at the sentence position of the title to which it belongs, multiple upper vector text sentences or / and multiple lower vector text sentences associated with the vector text sentence with the mask mark are selected and spliced ​​with the corresponding vector text sentence with the mask mark to generate an incremental vector text sentence to be input;

[0030] Performing mask prediction on each of the incremental vector text sentences to be input based on the Chinese general vocabulary by using the optimized large language model until a third proofread vector text sentence is output;

[0031] If the proofreading accuracy of any of the third proofread vector text sentences is greater than or equal to a preset accuracy threshold, generating a proofread ancient book text according to each of the third proofread vector text sentences;

[0032] If the collation accuracy of any of the third collated vector text sentences is less than a preset accuracy threshold, then selecting the second collated standard ancient book text for word segmentation and filtering low-frequency words according to the attribute information of the ancient book text to be collated, and constructing an optimized word list;

[0033] When any special word in the preset special word set does not exist in the optimized word list, the optimized word list is expanded according to the special word set to determine the target word list;

[0034] Using the optimized large language model to perform mask prediction on each of the incremental vector text sentences to be input based on the target vocabulary, until a fourth proofread vector text sentence is output;

[0035] A proofread ancient book text is generated based on each of the fourth proofread vector text sentences.

[0036] The second aspect of the present invention provides an ancient book text proofreading system based on a large language model, comprising:

[0037] A text acquisition module is used to acquire the text of the ancient book to be checked and proofread with mask marks;

[0038] A text preprocessing module, used for performing vector conversion on the text of the ancient book to be collated according to the title, and determining a plurality of vector text sentences of each title;

[0039] A mask prediction module, used to perform mask prediction on vector text sentences with mask tags based on a Chinese general vocabulary through a pre-trained large language model, and output a probability distribution of mask prediction words for each mask tag;

[0040] A text proofreading module, configured to proofread the associated vector text sentences with mask marks using the probability distribution of each mask prediction word, and output a corresponding first proofread vector text sentence;

[0041] The collation evaluation module is used to generate a collated ancient book text according to each of the first collated vector text sentences if the collation accuracy of any of the first collated vector text sentences is greater than or equal to a preset accuracy threshold.

[0042] A third aspect of the present invention provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the ancient book text collation method based on a large language model as described in any one of the above items.

[0043] A fourth aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed, implements the ancient book text collation method based on a large language model as described in any one of the above items.

[0044] A fifth aspect of the present invention provides a computer program product, comprising a computer program / instruction, which, when executed by a processor, implements the ancient book text collation method based on a large language model as described in any one of the above items.

[0045] It can be seen from the above technical solutions that the present invention has the following advantages:

[0046] The above scheme of the present invention provides a method for proofreading ancient book texts based on a large language model, including: obtaining the ancient book texts to be proofread with mask marks; performing vector conversion on the ancient book texts to be proofread according to the titles, and determining multiple vector text sentences of each title; performing mask prediction on the vector text sentences with mask marks based on the Chinese general vocabulary through a pre-trained large language model, and outputting the probability distribution of mask prediction words of each mask mark; using each mask prediction word probability distribution to proofread the associated vector text sentences with mask marks, and outputting the corresponding first proofread vector text sentences; if the proofreading accuracy of any first proofread vector text sentence is greater than or equal to the preset accuracy threshold, then generating the proofread ancient book text according to each first proofread vector text sentence. The above scheme uses the powerful context understanding and text prediction capabilities of the large language model to perform intelligent correction on the text that needs to be proofread, thereby improving the proofreading efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0048] Figure 1 A flowchart of a method for proofreading ancient books based on a large language model provided by an embodiment of the present invention;

[0049] Figure 2 A structural block diagram of an ancient book text collation system based on a large language model provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0050] The embodiment of the present invention provides a method and system for ancient book text collation based on a large language model, which is used to solve the technical problem that the current ancient book collation work still relies on expert experience and manually collates massive ancient book text data, resulting in low efficiency of ancient book collation.

[0051] In order to make the purpose, features and advantages of the present invention more obvious and easy to understand, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described below are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0052] See also Figure 1 , Figure 1A flowchart of the steps of an ancient book text collation method based on a large language model provided in an embodiment of the present invention.

[0053] The present invention provides an ancient book text proofreading method based on a large language model, comprising:

[0054] Step 101: Obtain the ancient book text to be proofread with mask marks.

[0055] It should be noted that the texts of ancient books to be proofread need to be proofread because they may contain word errors. Mask marks [MASK] are used to replace and mark the words that may contain errors. The marking process can be done by linguists and others manually reading the texts of ancient books to be proofread, or by using language models such as BERT to perform semantic understanding on the texts of ancient books to be proofread. On the one hand, they may be basic common sense errors, such as erroneous texts, missing texts, inverted texts, derivative texts, variant texts, and secondary variant texts in the digitization process. On the other hand, they may be derived through a probability-based analysis of the context, or they may be a combination of manual and model selection to determine the masked words that ultimately need to be marked.

[0056] Step 102: perform vector conversion on the ancient book text to be collated according to the title, and determine multiple vector text sentences for each title.

[0057] Step 102 includes the following sub-steps:

[0058] Standardize the punctuation marks of the ancient book text to be proofread and output the standardized ancient book text to be proofread;

[0059] Merge the text contents of different titles in the standardized ancient book text to be proofread with line breaks as boundaries to generate a converted ancient book text to be proofread;

[0060] After traversing and converting the text of the ancient book to be proofread in character units and removing the prohibited characters, vector conversion is performed to determine multiple vector text sentences of each title.

[0061] It should be noted that in order to facilitate the subsequent large language model to process text data more efficiently and accurately: first, the punctuation marks of the ancient book text to be collated are detected, and the punctuation marks are unified into the same style to obtain a standardized data text form that can be analyzed by the large language model, and the standardized ancient book text to be collated is obtained; then, the text content with splits and segments under the same title in the standardized ancient book text to be collated is merged and converted into one line. For example, for poetry texts, the verses under the same title are converted into one line, and the text content belonging to different titles is distinguished by line breaks, so as to obtain the converted ancient book text to be collated. In order to avoid the problem of missing or erroneous semantics in the text of ancient books to be collated, or the same sentence or paragraph being separated by line breaks during the division by title, which may lead to the original semantic loss and error in the subsequent text data conversion to vector, this method can remove the influence of extra line breaks. On the other hand, the text content of the same title can maintain good structural information, which makes it easier for the subsequent large language model to capture the contextual relationship of the text content. Furthermore, the converted text of ancient books to be collated is analyzed line by line, and the data content is traversed in units of characters. If there are characters in the preset banned character list, such as '__', '《', '》', '

', '

[0062] Step 103: Perform mask prediction on the vector text sentences with mask tags based on the Chinese general vocabulary through the pre-trained large language model, and output the probability distribution of mask prediction words for each mask tag.

[0063] Step 103 includes the following sub-steps:

[0064] When the vector text sentence with the mask mark is the first vector text sentence of the title to which it belongs, the vector text sentence with the mask mark is concatenated with the associated lower vector text sentence to obtain the vector text sentence to be input;

[0065] When the vector text sentence with the mask mark is the last vector text sentence of the title to which it belongs, the vector text sentence with the mask mark is concatenated with the associated upper vector text sentence to generate the vector text sentence to be input;

[0066] When the vector text sentence with the mask mark is the middle vector text sentence of the title to which it belongs, the vector text sentence with the mask mark is concatenated with the associated upper vector text sentence and lower vector text sentence to determine the vector text sentence to be input;

[0067] Each input vector text sentence is input into the pre-trained large language model in turn, and the corresponding output is the probability distribution of the masked predicted word of each mask tag with respect to the Chinese general vocabulary.

[0068] It should be noted that by using pre-trained large language models trained on a large number of ancient texts, such as SikuBERT and GuwenBERT, the model can be adapted to the distribution of ancient text data features; the pre-trained large language model is based on vector text sentences with mask markers [MASK] And the vector text sentence above Or / and vector text sentence Splicing is performed to generate a vector text sentence to be input as a model input. In a specific implementation, the vector text sentence to be input is selected according to the vector text sentence with a mask mark at the sentence position of the title to which it belongs, and the associated upper vector text sentence or / and lower vector text sentence is selected and spliced ​​with the corresponding vector text sentence with the mask mark, wherein when the vector text sentence with the mask mark is the first vector text sentence of the title to which it belongs, the associated lower vector text sentence is selected for splicing together, when the vector text sentence with the mask mark is the last vector text sentence of the title to which it belongs, the associated upper vector text sentence is selected for splicing together, and when the vector text sentence with the mask mark is the middle vector text sentence of the title to which it belongs, the associated upper vector text sentence and the lower vector text sentence are selected for joint splicing;

[0069] The vector text sentence to be input is input into the model for context analysis, and the [MASK] position is output about the Chinese general vocabulary The probability distribution of masked predicted words ,in , Represents the position of the vector text sentence with mask markers Masked predicted words The probability of occurrence of, it can be understood that the Chinese general vocabulary vocab is a vocabulary used for Chinese natural language processing (NLP) tasks, which contains commonly used Chinese words.

[0070] Step 104: Use the probability distribution of each masked predicted word to collate the associated vector text sentence with the mask mark, and output the corresponding first collated vector text sentence.

[0071] Step 104 includes the following sub-steps:

[0072] Based on the probability distribution of each mask prediction word, a plurality of mask prediction words with a preset number threshold are selected in descending order of probability, corresponding to the verification candidate word set constituting each mask mark;

[0073] If the masked word corresponding to the mask mark belongs to the associated verification candidate word set, the masked word is filled into the position of the mask mark, and the corresponding first proofread vector text sentence is output;

[0074] If the masked word corresponding to the mask mark does not belong to the associated verification candidate word set, the mask prediction word with the highest probability is selected from the verification candidate word set to fill the position of the mask mark, and the corresponding first proofread vector text sentence is output.

[0075] It should be noted that the preset number threshold with the highest probability is selected from the probability distribution of the mask prediction word to which each mask mark belongs. words and set them as the check candidate word set Then, according to the check candidate word set, the masked words that are missing or may be wrong are Conduct verification: When the masked word replaced by [MASK] exists in its verification candidate word set, it is considered that the word does not need to be proofread and corrected, and the original masked word is retained and filled in the position of the mask mark to output the corresponding first proofread vector text sentence; when the masked word replaced by [MASK] does not exist in its verification candidate word set, the mask prediction word with the highest probability is selected from its verification candidate word set to fill in the position of the mask mark to correct the masked word, thereby outputting the corresponding first proofread vector text sentence. It can be understood that for ancient texts, taking ancient poems as an example, some words have improvisational characteristics and cannot be proofread entirely by relying on the highest probability of association between words. By constructing a verification candidate word set to set a certain probability threshold window, the risk of misproofreading is reduced.

[0076] Step 105: If the proofreading accuracy of any first proofread vector text sentence is greater than or equal to a preset accuracy threshold, a proofread ancient book text is generated according to each first proofread vector text sentence.

[0077] It should be noted that in order to ensure the accuracy and reliability of the proofreading results, quality indicators are set for the first proofread vector text sentences obtained after proofreading for verification and evaluation. Taking the proofreading accuracy as an example of the quality indicator, when the proofreading accuracy is greater than or equal to the preset accuracy threshold, the proofreading quality is considered to meet the requirements. Thus, the proofread ancient book text is obtained according to the output of each first proofread vector text sentence.

[0078] The method also includes:

[0079] If the collation accuracy of any first collated vector text sentence is less than a preset accuracy threshold, the first collated standard ancient book text is selected according to the attribute information of the ancient book text to be collated to perform incremental learning on the pre-trained large language model, and an optimized large language model is output;

[0080] Using an optimized large language model to perform mask prediction on the vector text sentence with mask marks based on the Chinese general vocabulary, until a second proofread vector text sentence is output;

[0081] If the proofreading accuracy of any second proofread vector text sentence is greater than or equal to a preset accuracy threshold, a proofread ancient book text is generated according to each second proofread vector text sentence.

[0082] It should be noted that when the proofreading accuracy of the first proofread vector text sentence is less than the preset accuracy threshold, it is considered that the proofreading quality does not meet the requirements. Then, the model is retrained by performing incremental learning on the model based on the pre-trained large language model to adjust the training parameters to improve the overall proofreading quality: one or more first proofread standard ancient book texts are selected according to the attribute information of the ancient book text to be proofread. The attribute information may include stylistic attributes reflecting the literary genre and form of the ancient book text (such as Tang poetry and Song lyrics), time attributes reflecting the historical period or dynasty of the ancient book text (such as the Tang Dynasty and the Song Dynasty), and author attributes reflecting the creator of the ancient book text. After the proofread standard ancient book text that has been collated and has a strong correlation with the ancient book text to be proofread is cleaned according to the method of step 102, a random mask operation is performed to select 15%-20% of the words in the sentence to be covered and replaced with a mask mark [MASK] to obtain multiple incremental vector text sentences, and the model parameters are initialized, including the batch size (batch size). size) is set to 8, the initial learning rate is set to 5e-5 and the epoch is set to 10, and the pre-trained large language model is iteratively trained based on the incremental vector text sentence input in the manner of step 103, so that it can adapt to the data characteristics of the current specific ancient book text to be collated without forgetting the existing knowledge, thereby obtaining an optimized large language model;

[0083] After fine-tuning the model, the optimized large language model is used to restart the proofreading process based on the Chinese general vocabulary according to steps 103 and 104 to output the second proofread vector text sentence, and the proofreading accuracy of the second proofread vector text sentence is used to determine the proofreading quality. If the proofreading accuracy reaches the expected level, the proofread ancient book text is generated based on each second proofread vector text sentence.

[0084] The method also includes:

[0085] If the proofreading accuracy of any second proofread vector text sentence is less than a preset accuracy threshold, multiple upper vector text sentences or / and multiple lower vector text sentences associated with the vector text sentence with the mask mark are selected at the sentence position of the title to which the vector text sentence belongs, and the selected upper vector text sentences or / and the lower vector text sentences are spliced ​​with the corresponding vector text sentence with the mask mark to generate an incremental vector text sentence to be input;

[0086] By optimizing the large language model, mask prediction is performed on each incremental vector text sentence to be input based on the Chinese general vocabulary, until the third proofread vector text sentence is output;

[0087] If the proofreading accuracy of any third proofread vector text sentence is greater than or equal to a preset accuracy threshold, then generating a proofread ancient book text according to each third proofread vector text sentence;

[0088] If the collation accuracy of any third collated vector text sentence is less than the preset accuracy threshold, the second collated standard ancient book text is selected according to the attribute information of the ancient book text to be collated, and low-frequency words are filtered out to construct an optimized word list;

[0089] When any special word in the preset special word set does not exist in the optimized word list, the optimized word list is expanded according to the special word set to determine the target word list;

[0090] Using the optimized large language model to perform mask prediction on each incremental vector text sentence to be input based on the target vocabulary, until a fourth proofread vector text sentence is output;

[0091] A proofread ancient book text is generated based on each fourth proofread vector text sentence.

[0092] It should be noted that if the proofreading quality of the second proofread vector text sentence does not meet the quality expectations, consider increasing the input data size of the large language model by increasing the number of input sentences and expanding the context sentences to and , if the total tokens size is changed to 256 or 512 tokens (not exceeding the maximum length of the model input): when the vector text sentence with the mask mark is the first vector text sentence of the title to which it belongs, the vector text sentence with the mask mark is associated with multiple lower vector text sentences splicing to obtain the incremental vector text sentence to be input; when the vector text sentence with the mask mark is the last vector text sentence of the title to which it belongs, the vector text sentence with the mask mark is connected to the associated multiple upper vector text sentences Splicing is performed to generate incremental vector text sentences to be input; when the vector text sentence with the mask mark is the middle vector text sentence of the title to which it belongs, the vector text sentence with the mask mark is combined with the associated multiple upper vector text sentences and multiple down-vector text sentences splicing to determine the incremental vector text sentence to be input; then, re-collating the incremental vector text sentence to be input into the optimized large language model based on the Chinese general vocabulary until a third collated vector text sentence is output; if the collation quality of the third collated vector text sentence meets the quality requirements, determining the collated ancient book text based on the third collated vector text sentence;

[0093] If the collation quality of the third collated vector text sentence still does not meet the quality requirements, then consider optimizing the vocabulary: select one or more second collated standard ancient book texts according to the attribute information of the ancient book text to be collated. It can be understood that the second collated standard ancient book text and the first collated standard ancient book text may be the same or different; perform word segmentation on the second collated standard ancient book text to obtain multiple collated words, count the word frequency of each collated word, and eliminate the collated words whose word frequency is lower than the preset word frequency threshold to form an optimized vocabulary. By eliminating these words, the language style in the optimized vocabulary is better unified with the ancient book text to be collated, and the occurrence of interfering words is reduced, thereby optimizing the probability distribution of each word in the vocabulary. In the optimized vocabulary, the collated words can be sorted from large to small according to the word frequency, and each collated word is aligned with the to-be-collated ancient book text. The encoding, that is, the digital ID forms a mapping relationship; since special words such as '[PAD]', '[UNK]', '[CLS]', '[SEP]' and '[MASK]' play a specific role in the mask prediction of the large language model, a preset special word set is set to determine whether multiple special words in the set exist in the optimized word list. When any special word in the preset special word set exists in the optimized word list, the optimized word list is determined as the target word list, otherwise the optimized word list is expanded according to the special word set, so as to be determined as the target word list; then, each incremental vector text sentence to be input is input into the optimized large language model in turn for mask prediction based on the target word list until the fourth proofread vector text sentence is output, and each fourth proofread vector text sentence is used to construct the proofread ancient book text.

[0094] In the embodiment of the present invention, the powerful context understanding and text prediction capabilities of the large language model are used to perform intelligent correction on the text that needs to be proofread, thereby improving the proofreading efficiency, and further improving the efficiency of ancient book digitization and restoration. At the same time, the model fine-tuning, input parameter optimization and field data enhancement of the ancient book text to be proofread are combined to optimize the text proofreading quality.

[0095] See also Figure 2 , Figure 2 A structural block diagram of an ancient book text collation system based on a large language model provided in an embodiment of the present invention.

[0096] The present invention provides an ancient book text proofreading system based on a large language model, comprising:

[0097] The text acquisition module 201 is used to acquire the ancient book text to be checked and proofread with mask marks;

[0098] The text preprocessing module 202 is used to perform vector conversion on the ancient book text to be collated according to the title, and determine multiple vector text sentences for each title;

[0099] The mask prediction module 203 is used to perform mask prediction on the vector text sentences with mask tags based on the Chinese general vocabulary through the pre-trained large language model, and output the probability distribution of mask prediction words for each mask tag;

[0100] A text proofreading module 204, configured to proofread the associated vector text sentence with the mask mark using the probability distribution of each mask prediction word, and output a corresponding first proofread vector text sentence;

[0101] The collation evaluation module 205 is used to generate collated ancient book texts according to each of the first collated vector text sentences if the collation accuracy of any of the first collated vector text sentences is greater than or equal to a preset accuracy threshold.

[0102] Optionally, the text preprocessing module 202 includes:

[0103] A data loading unit is used to standardize the punctuation marks of the ancient book text to be collated and output the standardized ancient book text to be collated;

[0104] A data segmentation unit is used to merge the text contents of different titles in the standardized ancient book text to be collated and proofread using line breaks as boundaries to generate a converted ancient book text to be collated and proofread;

[0105] The data conversion unit is used to traverse and convert the text of the ancient book to be proofread in units of characters, remove the prohibited characters, and then perform vector conversion to determine multiple vector text sentences of each title.

[0106] Optionally, the mask prediction module 203 is specifically used for:

[0107] When the vector text sentence with the mask mark is the first vector text sentence of the title to which it belongs, the vector text sentence with the mask mark is concatenated with the associated lower vector text sentence to obtain the vector text sentence to be input;

[0108] When the vector text sentence with the mask mark is the last vector text sentence of the title to which it belongs, the vector text sentence with the mask mark is concatenated with the associated upper vector text sentence to generate the vector text sentence to be input;

[0109] When the vector text sentence with the mask mark is the middle vector text sentence of the title to which it belongs, the vector text sentence with the mask mark is concatenated with the associated upper vector text sentence and lower vector text sentence to determine the vector text sentence to be input;

[0110] Each input vector text sentence is input into the pre-trained large language model in turn, and the corresponding output is the probability distribution of the masked predicted word of each mask tag with respect to the Chinese general vocabulary.

[0111] Optionally, the text proofreading module 204 is specifically used for:

[0112] Based on the probability distribution of each mask prediction word, a plurality of mask prediction words with a preset number threshold are selected in descending order of probability, corresponding to the verification candidate word set constituting each mask mark;

[0113] If the masked word corresponding to the mask mark belongs to the associated verification candidate word set, the masked word is filled into the position of the mask mark, and the corresponding first proofread vector text sentence is output;

[0114] If the masked word corresponding to the mask mark does not belong to the associated verification candidate word set, the mask prediction word with the highest probability is selected from the verification candidate word set to fill the position of the mask mark, and the corresponding first proofread vector text sentence is output.

[0115] Optionally, a re-calibration module is also included, for:

[0116] If the collation accuracy of any first collated vector text sentence is less than a preset accuracy threshold, the first collated standard ancient book text is selected according to the attribute information of the ancient book text to be collated to perform incremental learning on the pre-trained large language model, and an optimized large language model is output;

[0117] Using an optimized large language model to perform mask prediction on the vector text sentence with mask marks based on the Chinese general vocabulary, until a second proofread vector text sentence is output;

[0118] If the proofreading accuracy of any second proofread vector text sentence is greater than or equal to a preset accuracy threshold, a proofread ancient book text is generated according to each second proofread vector text sentence.

[0119] Optionally, the re-collation module is further configured to:

[0120] If the proofreading accuracy of any second proofread vector text sentence is less than a preset accuracy threshold, multiple upper vector text sentences or / and multiple lower vector text sentences associated with the vector text sentence with the mask mark are selected at the sentence position of the title to which the vector text sentence belongs, and the selected upper vector text sentences or / and the lower vector text sentences are spliced ​​with the corresponding vector text sentence with the mask mark to generate an incremental vector text sentence to be input;

[0121] By optimizing the large language model, mask prediction is performed on each incremental vector text sentence to be input based on the Chinese general vocabulary, until the third proofread vector text sentence is output;

[0122] If the proofreading accuracy of any third proofread vector text sentence is greater than or equal to the preset accuracy threshold, then generating a proofread ancient book text according to each third proofread vector text sentence;

[0123] If the collation accuracy of any third collated vector text sentence is less than the preset accuracy threshold, the second collated standard ancient book text is selected according to the attribute information of the ancient book text to be collated, and low-frequency words are filtered out to construct an optimized word list;

[0124] When any special word in the preset special word set does not exist in the optimized word list, the optimized word list is expanded according to the special word set to determine the target word list;

[0125] Using the optimized large language model to perform mask prediction on each incremental vector text sentence to be input based on the target vocabulary, until a fourth proofread vector text sentence is output;

[0126] A proofread ancient book text is generated based on each fourth proofread vector text sentence.

[0127] An embodiment of the present invention also provides a computer device, including a memory and a processor, wherein a computer program is stored in the memory; when the computer program is executed by the processor, the processor executes the steps of the ancient book text collation method based on a large language model as described in any of the above embodiments.

[0128] An embodiment of the present invention further provides a computer-readable storage medium having a computer program / instruction stored thereon. When the computer program / instruction is executed by a processor, the steps of the ancient book text collation method based on a large language model as in any of the above embodiments are implemented.

[0129] An embodiment of the present invention further provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of the ancient book text collation method based on a large language model as in any of the above embodiments.

[0130] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above-described system and module can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0131] In the several embodiments provided in the present application, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0132] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0133] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0134] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0135] As described above, the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features thereof may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for proofreading ancient books based on a large language model, characterized in that: include: Obtain the ancient book text to be proofread with mask marks; Performing vector conversion on the ancient book text to be collated according to the title, and determining a plurality of vector text sentences of each title; Performing mask prediction on vector text sentences with mask tags based on a Chinese general vocabulary using a pre-trained large language model, and outputting a probability distribution of mask prediction words for each mask tag; Using each of the mask prediction word probability distributions to collate the associated vector text sentence with the mask mark, and outputting a corresponding first collated vector text sentence; If the collation accuracy of any of the first collated vector text sentences is greater than or equal to a preset accuracy threshold, a collated ancient book text is generated according to each of the first collated vector text sentences.

2. The ancient book text proofreading method based on a large language model according to claim 1 is characterized in that: The step of performing vector conversion on the text of the ancient book to be collated and proofread according to the title, and determining a plurality of vector text sentences of each title, comprises: Standardizing the punctuation marks of the ancient book text to be proofread and outputting the standardized ancient book text to be proofread; Merging the text contents of different titles in the standardized ancient book text to be proofread with line breaks as boundaries to generate a converted ancient book text to be proofread; After traversing the ancient book text to be collated and proofread in character units and removing prohibited characters, vector conversion is performed to determine a plurality of vector text sentences of each of the titles.

3. The ancient book text proofreading method based on a large language model according to claim 1 is characterized in that: The method performs mask prediction on the vector text sentences with mask marks based on the Chinese general vocabulary by using the pre-trained large language model, and outputs the probability distribution of mask prediction words of each mask mark, including: When the vector text sentence with the mask mark is the first vector text sentence of the title to which it belongs, the vector text sentence with the mask mark is concatenated with the associated lower vector text sentence to obtain the vector text sentence to be input; When the vector text sentence with the mask mark is the last vector text sentence of the title to which it belongs, the vector text sentence with the mask mark is concatenated with the associated upper vector text sentence to generate the vector text sentence to be input; When the vector text sentence with the mask mark is an intermediate vector text sentence of the title to which it belongs, the vector text sentence with the mask mark is concatenated with the associated upper vector text sentence and lower vector text sentence to determine the vector text sentence to be input; Each of the input vector text sentences is sequentially input into the pre-trained large language model, and the probability distribution of the masked predicted words of each masked tag with respect to the Chinese general vocabulary is correspondingly output.

4. The ancient book text proofreading method based on a large language model according to claim 1 is characterized in that: The step of using each of the mask prediction word probability distributions to collate the associated vector text sentence with the mask mark and outputting the corresponding first collated vector text sentence includes: Based on the probability distribution of each of the mask prediction words, a plurality of mask prediction words with a preset number threshold are selected in descending order of probability, corresponding to the verification candidate word sets constituting each of the mask marks; If the masked word corresponding to the mask mark belongs to the associated verification candidate word set, the masked word is filled into the position of the mask mark, and the corresponding first proofread vector text sentence is output; If the masked word corresponding to the mask mark does not belong to the associated verification candidate word set, the mask prediction word with the highest probability is selected from the verification candidate word set to fill the position of the mask mark, and the corresponding first proofread vector text sentence is output.

5. The ancient book text proofreading method based on a large language model according to claim 1 is characterized in that: Also includes: If the collation accuracy of any of the first collated vector text sentences is less than a preset accuracy threshold, selecting a first collated standard ancient book text according to the attribute information of the ancient book text to be collated to perform incremental learning on the pre-trained large language model, and outputting an optimized large language model; Using the optimized large language model to perform mask prediction on the vector text sentence with the mask mark based on the Chinese general vocabulary, until a second proofread vector text sentence is output; If the proofreading accuracy of any of the second proofread vector text sentences is greater than or equal to a preset accuracy threshold, a proofread ancient book text is generated according to each of the second proofread vector text sentences.

6. The ancient book text proofreading method based on a large language model according to claim 5 is characterized in that: Also includes: If the proofreading accuracy of any of the second proofread vector text sentences is less than a preset accuracy threshold, then according to the vector text sentence with the mask mark at the sentence position of the title to which it belongs, multiple upper vector text sentences or / and multiple lower vector text sentences associated with the vector text sentence with the mask mark are selected and spliced ​​with the corresponding vector text sentence with the mask mark to generate an incremental vector text sentence to be input; Performing mask prediction on each of the incremental vector text sentences to be input based on the Chinese general vocabulary by using the optimized large language model until a third proofread vector text sentence is output; If the proofreading accuracy of any of the third proofread vector text sentences is greater than or equal to a preset accuracy threshold, generating a proofread ancient book text according to each of the third proofread vector text sentences; If the collation accuracy of any of the third collated vector text sentences is less than a preset accuracy threshold, then selecting the second collated standard ancient book text for word segmentation and filtering low-frequency words according to the attribute information of the ancient book text to be collated, and constructing an optimized word list; When any special word in the preset special word set does not exist in the optimized word list, the optimized word list is expanded according to the special word set to determine the target word list; Using the optimized large language model to perform mask prediction on each of the incremental vector text sentences to be input based on the target vocabulary, until a fourth proofread vector text sentence is output; A proofread ancient book text is generated based on each of the fourth proofread vector text sentences.

7. An ancient book text proofreading system based on a large language model, characterized in that: include: A text acquisition module is used to acquire the text of the ancient book to be checked and proofread with mask marks; A text preprocessing module, used for performing vector conversion on the text of the ancient book to be collated according to the title, and determining a plurality of vector text sentences of each title; A mask prediction module, used to perform mask prediction on vector text sentences with mask tags based on a Chinese general vocabulary through a pre-trained large language model, and output a probability distribution of mask prediction words for each mask tag; A text proofreading module, configured to proofread the associated vector text sentences with mask marks using the probability distribution of each mask prediction word, and output a corresponding first proofread vector text sentence; The collation evaluation module is used to generate a collated ancient book text according to each of the first collated vector text sentences if the collation accuracy of any of the first collated vector text sentences is greater than or equal to a preset accuracy threshold.

8. A computer device, characterized in that: It comprises a memory and a processor, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the processor executes the steps of the ancient book text collation method based on a large language model as described in any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program / instruction stored thereon, characterized in that: When the computer program / instructions are executed by a processor, the steps of the ancient book text collation method based on a large language model as described in any one of claims 1 to 6 are implemented.

10. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps of the ancient book text collation method based on a large language model as described in any one of claims 1 to 6 are implemented.

Images