A text processing method, system, terminal, and storage medium based on a large model.

Abstract

This invention belongs to the field of text processing technology, specifically relating to a text processing method, system, terminal, and storage medium based on a large-scale model. The method includes preprocessing the input text, inputting the preprocessed text into a pre-built first large-scale model, and outputting an error list or a no-error message. While the first large-scale model outputs the error list, the method detects the error list and corrects the preprocessed text based on the detected error list. The corrected text or the text corresponding to the no-error message is then input into a pre-built second large-scale language model, which outputs a sensitive word message or a no-sensitive-word message. This invention proactively identifies and handles sensitive word issues that may be introduced or triggered by error correction through a sequential process of error correction followed by sensitive word analysis. The second large-scale model performs sensitive word analysis based on the corrected text, significantly improving the real-time accuracy and security of sensitive word identification.

Description

Technical Field

[0001] This invention belongs to the field of text processing technology, specifically relating to a text processing method, system, terminal, and storage medium based on a large model. Background Technology

[0002] In today's digital age, the demand for text data processing is growing, especially in text correction and sensitive word analysis. Traditional methods have many limitations in handling these issues. For example, rule-based methods rely on manually defined grammar rules and dictionaries, making it difficult to adapt to the diversity and complexity of languages, especially when dealing with emerging vocabulary and trending internet terms, where rule maintenance costs are extremely high. Traditional machine learning methods, such as Hidden Markov Models (HMMs) and Conditional Random Fields (CRFs), while capable of handling some simple text features, perform poorly in dealing with long-distance dependencies and complex semantics, and have poor generalization ability to specialized terminology and emerging vocabulary.

[0003] Furthermore, traditional methods are inefficient when processing large-scale text streams, making it difficult to meet real-time requirements. More importantly, in existing technologies, text correction and sensitive word analysis are usually run as independent modules, ignoring the potential for sensitive words to appear in the corrected text, leading to insufficient accuracy in sensitive word identification. For example, some words may be given new meanings after correction, thus becoming sensitive words, and traditional methods cannot effectively identify such changes. Summary of the Invention

[0004] To address the shortcomings of existing technologies where text correction and sensitive word analysis are typically run as independent modules, neglecting potential sensitive word issues in the corrected text and resulting in insufficient accuracy in sensitive word recognition, this invention provides a text processing method, system, terminal, and storage medium based on a large model to solve the aforementioned technical problems.

[0005] In a first aspect, the present invention provides a text processing method based on a large model, comprising:

[0006] Step S1: Preprocess the input text, which includes extracting text, word segmentation, normalization, and paragraph segmentation in sequence.

[0007] Step S2: Input the preprocessed text into the pre-built first large model, output an error list or an error-free message; and when the first large model outputs an error list, detect the output error list and correct the preprocessed text based on the detected error list.

[0008] Step S3: Input the corrected text or the text corresponding to the error-free prompt into the pre-built second language model, and output sensitive word prompts or no sensitive word prompts;

[0009] Step S4: Merge the output results of the first pre-built large model and the output results of the second pre-built large language model; format the merged output results, output the correction document, and clean up the correction document after outputting the correction document.

[0010] Further improvements to this technical solution include step S1, which includes:

[0011] Determine if the input text is not a plain text document; if so, call the document parsing interface of the pre-stored MarkitDown tool to extract the text content; otherwise, directly read the text content of the input text; non-plain text documents include PDF documents and Word documents.

[0012] The extracted text content is segmented by using a pre-stored word segmentation algorithm and a pre-defined dictionary.

[0013] The segmented text content is standardized, including unifying capitalization, correcting spelling errors, converting number symbols, removing redundant whitespace and punctuation, stemming, and restoring word forms.

[0014] The text content after normalization is processed to identify line breaks and encode adjacent sentences, and the semantic relevance of adjacent sentences is calculated; when the semantic relevance of adjacent sentences is lower than a set threshold, it is divided into a new paragraph.

[0015] The text content after being divided into paragraphs is formed into preprocessed text.

[0016] Further improvements to this technical solution include the following method for pre-building the first major model in step S2:

[0017] Historical text error correction annotation data is collected and organized into triples containing the original text, error location, and correction content. An error correction training set is then constructed based on the organized triples. and error correction verification set ;

[0018] Based on the first pre-trained large model, the main parameters are frozen, and training cue vectors are inserted into the input layer of the first pre-trained large model. ,in, To indicate the length, The hidden layer dimension of the first pre-trained large model;

[0019] The first pre-trained large model uses a configuration-based loss function. Training is performed, where N is the error correction training set. The number of samples in For the first The incorrect label of a sample This is the inference function of the first pre-trained large model. For input text, This is a vector concatenation operation;

[0020] In the error correction verification set Calculate the error correction accuracy Recall rate ,satisfy and When that happens, save the current model as the first major model.

[0021] Further improvements to this technical solution include the following method for detecting the output error list in step S2:

[0022] According to the formula Calculate the consistency score of the error list ;in, For multiple rounds of reasoning; This is a variant of the error list for the j-th inference; For pre-stored text semantic similarity functions;

[0023] The consistency score of the calculated error list With the preset self-consistency score threshold Compare;

[0024] like If a false alarm is detected in the error list, it will be filtered out, and only the false alarm will be retained. Error entries, update the error list to .

[0025] Further improvements to this technical solution include the following method for pre-building the second major model in step S3:

[0026] Collect sensitive word data and assign category labels to each sensitive word. ; and organized into a sensitive word database by category. and for sensitive word database Each sensitive word in the text is marked with a risk level. ;

[0027] Based on the second pre-trained large language model, the main parameters are frozen, and a sensitive word library embedding matrix is ​​injected into the input layer of the second pre-trained large language model. ;in, For sensitive word database The total number of sensitive words in the database, i.e., the sensitive word database The number of elements in the middle; The dimension of the embedded vector;

[0028] Based on the compiled sensitive word database Constructing a sensitive word training set and sensitive word verification set ;

[0029] The second pre-trained large model uses a configuration-based loss function. Training was conducted; among them, Training set for sensitive words The number of labeled samples in the data; Training set for sensitive words The Middle Sensitive word labels for each sample ; This is the inference function for the second pre-trained large model; This is the text corresponding to the correction text or error-free message; This is a vector concatenation operation;

[0030] In the sensitive word verification set The accuracy of sensitive word identification is calculated above. Recall rate ,satisfy and When that happens, save the current model as the second largest model.

[0031] Further improvements to this technical solution include the following method for generating sensitive word suggestions in step S3:

[0032] The second model detects sensitive words. Extract its position in the correction text or the text corresponding to the error-free message. Risk level Generate sensitive word suggestions ;

[0033] Sensitive word alerts Concatenate with the text context to calculate the association score. : ;in, The second largest model determines the probability that "sensitive words are context-dependent"; and when When this happens, it is identified as a "false positive sensitive word" and filtered.

[0034] Further improvements to this technical solution include step S4, which includes:

[0035] The error list output by the first major model Sensitive word suggestions from the second largest model output Parsing JSON data into structured data; where, This indicates the location of the text error. For text error types; For the revised content;

[0036] Merge the parsed results in ascending order of text position coordinates to generate a unified result list: ;in, This is a list of the merged results; This is a set merging operation; This is a function for sorting by position.

[0037] Secondly, the present invention provides a text processing system based on a large model, comprising:

[0038] The text preprocessing module is used to preprocess the input text. The preprocessing includes text extraction, word segmentation, normalization, and paragraph segmentation, which are performed sequentially.

[0039] The text correction module is used to input the pre-processed text into the pre-built first model, output an error list or an error-free message; and when the first model outputs an error list, it detects the output error list and corrects the pre-processed text based on the detected error list.

[0040] The sensitive word analysis module is used to input the corrected text or the text corresponding to the error-free prompt into the pre-built second language model and output sensitive word prompts or no sensitive word prompts.

[0041] The results processing module is used to merge the output results of the pre-built first large model and the output results of the pre-built second large language model; format the merged output results; output a correction document; and clean up the correction document after outputting the correction document.

[0042] Thirdly, the present invention provides a terminal, comprising:

[0043] Processor, memory, among which,

[0044] This memory is used to store computer programs.

[0045] The processor is used to retrieve and run the computer program from memory, causing the terminal to perform the terminal method described above.

[0046] Fourthly, the present invention provides a computer storage medium storing instructions that, when executed on a computer, cause the computer to perform the methods described in the above aspects.

[0047] The beneficial effects of this invention are as follows:

[0048] By utilizing pre-trained large language models instead of traditional rule-based or shallow machine learning models, both the first and second large models possess powerful semantic understanding and context modeling capabilities. This enables the system to effectively handle the diversity and complexity of language, including emerging vocabulary, internet slang, technical terms, and texts containing long-distance dependencies and complex semantics, significantly improving the accuracy and generalization ability of text correction and sensitive word identification. Through the autonomous learning capabilities of the large models, the reliance on manually customized rules and dictionaries is greatly reduced, effectively overcoming the problems of high rule maintenance costs and poor adaptability in traditional methods.

[0049] By employing a sequential process of error correction followed by sensitive word analysis, the system proactively identifies and addresses sensitive word issues that may be introduced or triggered during error correction. The second model performs sensitive word analysis based on the corrected text, ensuring that the context for sensitive word identification is based on the most accurate text content. This completely resolves the critical flaw of "ignoring potential sensitive word issues in the corrected text," significantly improving the real-time accuracy and security of sensitive word identification.

[0050] In the text correction stage, an error list self-consistency detection mechanism was added. This mechanism effectively identifies and filters false positives that may be generated by the first model through multiple inferences and semantic similarity comparisons, ensuring that the final error list used for text correction has higher reliability and improving the precision and credibility of the correction results.

[0051] In the sensitive word analysis stage, by calculating the correlation score between sensitive word prompts and context, and based on the threshold-based filtering mechanism, it is possible to effectively identify and exclude "falsely reported sensitive words" that are harmless due to context, which greatly reduces the false alarm rate of sensitive word identification and makes the results more practically valuable. Attached Figure Description

[0052] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0053] Figure 1 This is a schematic flowchart illustrating a method according to an embodiment of the present invention.

[0054] Figure 2 This is a schematic block diagram of a system according to an embodiment of the present invention.

[0055] Figure 3 This is a schematic diagram of the structure of a terminal provided in an embodiment of the present invention.

[0056] 210 is the text preprocessing module, 220 is the text correction module, 230 is the sensitive word analysis module, and 240 is the result processing module. Detailed Implementation

[0057] To make the objectives, features, and advantages of this invention more apparent and understandable, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings of the specific embodiments. Obviously, the embodiments described below are only some embodiments of this invention, and not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0058] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein in the description of the invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention.

[0059] Figure 1 This is a schematic flowchart illustrating a text processing method based on a large model provided by the present invention. Wherein, Figure 1 The executing entity can be a text processing system based on a large model. Depending on different requirements, the order of steps in this flowchart can be changed, and some can be omitted.

[0060] like Figure 1 As shown, the method includes:

[0061] Step S1: Preprocess the input text, which includes extracting text, word segmentation, normalization, and paragraph segmentation in sequence.

[0062] Step S2: Input the preprocessed text into the pre-built first large model, output an error list or an error-free message; and when the first large model outputs an error list, detect the output error list and correct the preprocessed text based on the detected error list.

[0063] Step S3: Input the corrected text or the text corresponding to the error-free prompt into the pre-built second language model, and output sensitive word prompts or no sensitive word prompts;

[0064] Step S4: Merge the output results of the first pre-built large model and the output results of the second pre-built large language model; format the merged output results, output the correction document, and clean up the correction document after outputting the correction document.

[0065] This invention proactively identifies and handles sensitive word issues that may be introduced or triggered by error correction through a sequential process of first correcting errors and then analyzing sensitive words. The second model performs sensitive word analysis based on the corrected text, ensuring that the context of sensitive word identification is based on the most accurate text content. This completely solves the key deficiency of "ignoring sensitive word issues that may appear in the text after error correction," significantly improving the real-time accuracy and security of sensitive word identification.

[0066] To facilitate understanding of the present invention, the following description further illustrates the text processing method based on a large model provided by the present invention, using the principle of the large model-based text processing method and the process of text processing based on a large model in the embodiments.

[0067] Step S1 includes:

[0068] Determine if the input text is not a plain text document; if so, call the document parsing interface of the pre-stored MarkitDown tool to extract the text content; otherwise, directly read the text content of the input text; non-plain text documents include PDF documents and Word documents.

[0069] The extracted text content is segmented by using a pre-stored word segmentation algorithm and a pre-defined dictionary.

[0070] The segmented text content is standardized, including unifying capitalization, correcting spelling errors, converting number symbols, removing redundant whitespace and punctuation, stemming, and restoring word forms.

[0071] The text content after normalization is processed to identify line breaks and encode adjacent sentences, and the semantic relevance of adjacent sentences is calculated; when the semantic relevance of adjacent sentences is lower than a set threshold, it is divided into a new paragraph.

[0072] The text content after being divided into paragraphs is formed into preprocessed text.

[0073] The type of input text is determined by the file extension (e.g., .pdf, .docx, .txt). If the extension is .pdf or .docx, it is considered a non-plain text document; if it is .txt, it is considered a plain text document.

[0074] Call the parsing interfaces of pandoc (a cross-format document conversion tool) or Python libraries (PyPDF2 for PDF and python-docx for Word) to extract the text content. For example, use the document.paragraphs interface of python-docx to read the paragraphs of a Word document and extract the text of PDF pages through the pages interface of PyPDF2.PdfReader, ensuring that the paragraph order and basic format (such as line breaks and list symbols) of the original text are retained.

[0075] For.txt files, directly read the text content through a file stream (such as the open() function in Python) to avoid redundant format parsing operations.

[0076] Adopt Jieba分词 (for Chinese) or word_tokenize of NLTK (for English) as the pre-stored word segmentation algorithm to support the processing of mixed Chinese and English texts.

[0077] Construct a preset dictionary that includes common vocabulary (such as "artificial intelligence", "data processing"), domain terms (such as "limitation of action" in the legal field and "targeted therapy" in the medical field), and stop words (such as "的", "了", "and"). When performing word segmentation, prioritize matching the words in the preset dictionary, and split the unmatched items with the highest probability (for example, "自然语言处理" is matched as "自然语言" + "处理" according to the dictionary instead of single character segmentation).

[0078] Split the extracted text content by sentence groups (ending with punctuation marks "。?!;"), and then perform word segmentation on each sentence group, outputting a list of word segmentation results (such as "我爱文本处理" → ["我", "爱", "文本处理"]).

[0079] Unify case: Convert English words to lowercase (such as "TextProcessing" → "textprocessing"), and keep Chinese as it is (no case distinction).

[0080] Correct text spelling mistakes: Call the pyenchant (for English) or pinyin (for Chinese) library to perform spelling verification on the word segmentation results. For example, correct "teh" to "the" and "相象" to "想象".

[0081] Convert digital symbol formats: Convert Roman numerals to Arabic numerals (such as "Ⅲ" → "3"), Chinese numerals to Arabic numerals (such as "五十" → "50"), and unify the date format (such as "2023年10月" → "2023-10").

[0082] Remove redundant whitespace and punctuation: Delete consecutive spaces (retain single spaces), meaningless line breaks, unify punctuation to half-width format (e.g., “,” → “,”, “;” → “;”), and remove redundant punctuation at the beginning and end of the text (e.g., “

text content

[0083] Stem extraction and lemmatization: Perform stem extraction on English words (e.g., “running” → “run”), and lemmatization on Chinese words (e.g., “处理过” → “处理”).

[0084] Line break recognition: Initially identify the boundaries of the original paragraphs by detecting the \n or \r\n markers in the text.

[0085] Sentence encoding and semantic relatedness calculation: Convert the normalized sentence groups into vectors (using Word2Vec or BERT embeddings), and calculate the semantic relatedness of adjacent sentences through the cosine similarity function Sim(·,·) (the value range is [0,1]). For example, the relatedness between the sentences “The weather is sunny today” and “Suitable for outdoor activities” is relatively high (about 0.8), and the relatedness with “Machine learning is a branch of artificial intelligence” is relatively low (about 0.2).

[0086] Paragraph division threshold: Set the semantic relatedness threshold to 0.3 (which can be dynamically adjusted). When the relatedness of adjacent sentences ≥0.3, they are merged into the same paragraph; when the relatedness <0.3, they are divided into new paragraphs. For example, the relatedness between “Meeting time: October 2023” and “Participants: Zhang and Li” is low (<0.3), so they are divided into two paragraphs; the relatedness between “The meeting discussed the project progress” and “Determined the next step plan” is high (>0.3), so they are merged into one paragraph.

[0087] Integrate the text after paragraph division in the original order to form a preprocessed text with a clear structure, in the format: [Paragraph 1 text]\n[Paragraph 2 text]\n... (\n is the paragraph separator), which is used as the input to the text error correction module.

[0088] In addition, the method for pre-building the first large model in step S2 includes:

[0089] Collect historical text error correction annotation data, organize it in the format of a triple containing the original text, error location, and corrected content, and construct an error correction training set according to the content of the sorted triples and an error correction validation set ;

[0090] Based on the first pre-trained large model, freeze the main parameters, and insert a training prompt vector into the input layer of the first pre-trained large model , where is the prompt length, is the hidden layer dimension of the first pre-trained large model;

[0091] The first pre-trained large model uses a configuration-based loss function. Training is performed, where N is the error correction training set. The number of samples in For the first The incorrect label of a sample This is the inference function of the first pre-trained large model. For input text, This is a vector concatenation operation;

[0092] In the error correction verification set Calculate the error correction accuracy Recall rate ,satisfy and When that happens, save the current model as the first major model.

[0093] We collect historical texts from various fields, including news reports, social media comments, and official documents, covering both general scenarios and professional fields (such as medicine and law), to ensure data diversity.

[0094] Annotation rules: Through a combination of manual annotation and machine pre-annotation, the error locations (accurate to character index), error types (such as spelling errors, grammatical errors, semantic ambiguities, etc.), and corrections in the original text are annotated, forming a triple format: (original text, list of error locations, list of corrections). For example, the triple of the original text "He is going to school tomorrow" is: ("He is going to school tomorrow", [(4,5)], ["again"]) ("at" is corrected to "again", the error location is the 4th-5th character).

[0095] The triplet sets were divided into error correction training sets at an 8:2 ratio. and error correction verification set The training set contains no fewer than 100,000 samples, and the validation set covers typical error types across various fields.

[0096] We select a pre-trained large model with strong semantic understanding capabilities as the foundation, such as BERT-base (suitable for general scenarios) or MedicalBERT (suitable for the medical field), with a hidden layer dimension d of 768 (BERT-base).

[0097] Insert trainable cue vectors into the model input layer The prompt length m is set to 10 (which can be dynamically adjusted according to the text length), and the vector is initialized with a normally distributed random value. It is concatenated with the input text vector and then input into the model. The purpose of the prompt vector is to guide the model to focus on the "error correction task". For example, the task orientation is strengthened by semantic embedding of "Please detect and correct the error in the following text: [text]".

[0098] 90% of the main parameters of the base model were frozen (only the top Transformer layer and cue vectors were fine-tuned), the Adam optimizer was used, the learning rate was set to 5e-5, the number of samples was 32, the number of training epochs was 10, and the performance was evaluated on the validation set at the end of each epoch.

[0099] In the error correction verification set The above calculations are as follows: Error Correction Acc is the sum of the number of correctly corrected errors and the total number of errors predicted by the first-largest model; Error Correction Recall is the sum of the number of correctly corrected errors and the total number of errors actually present in the text. When the validation set satisfies... and When the time comes, stop training, save the current model parameters, and make the current model the first major model.

[0100] Preprocessed input text (After word segmentation and normalization) the data is input into the first large model, which then performs error prediction distribution. ;in, For text The length of the text; Size of the error type vocabulary.

[0101] Set an error probability threshold The value is 0.85 (which can be dynamically adjusted according to the domain), and the criteria are filtered to meet the requirements. Lexical units: error position (Record the index of the word in the text) and error type (Record the corresponding error type, such as "grammar error - misuse of preposition").

[0102] Furthermore, the method for detecting the output error list in step S2 includes:

[0103] According to the formula Calculate the consistency score of the error list ;in, For multiple rounds of reasoning; This is a variant of the error list for the j-th inference; For pre-stored text semantic similarity functions;

[0104] The consistency score of the calculated error list With the preset self-consistency score threshold Compare;

[0105] like If a false alarm is detected in the error list, it will be filtered out, and only the false alarm will be retained. Error entries, update the error list to .

[0106] When the first large model outputs the initial error list Then, a multi-round reasoning mechanism is initiated, and the number of reasoning rounds is set. (This can be dynamically adjusted based on text complexity; for complex text, select...) ).

[0107] Different inference conditions can be generated by adjusting the input prompt template, for example:

[0108] Tip 1: "Please check the text for grammatical errors, including inappropriate word choice and word order inversion."

[0109] Tip 2: "Please focus on identifying semantic ambiguities in the text and ignore technical terms."

[0110] Tip 3: "Please check for errors that may cause misunderstanding from the reader's perspective";

[0111] Each inference outputs a variant of the error list. The variant content includes the error location, type, and suggested fixes, and is formatted the same as the initial error list.

[0112] Calculate the initial error list and text semantic similarity function based on multi-round variants. With the list of language errors variants Calculate the consistency score of the error list. The default self-consistency score threshold is 0.7 (general), which can be increased to 0.75 in professional fields (such as law and medicine). Optimize through validation set testing (to ensure that the self-consistency score of more than 90% of real errors is greater than or equal to the threshold).

[0113] Score comparison logic: Calculate the self-consistency score of the initial error list. Then, it is compared with the preset self-consistency score threshold. Compare them.

[0114] When the overall score is below the threshold, each error item in the initial error list is individually validated: the score of that error item is counted. Number of times in each variant (For example, if a certain error occurs 3 times in 5 variations,) ); Calculate the support rate for a single error item ,reserve Errors (i.e., those appearing in at least 60% of the variants, considered genuine errors) are filtered out. False alarms (such as technical terms being misjudged as errors).

[0115] The filtered errors are consolidated into a new error list. The format is the same as the initial list, and it serves as the error message in the final output to the result processing module.

[0116] In the text correction stage, this invention adds an error list self-consistency detection mechanism. This mechanism, through multiple inferences and semantic similarity comparisons, effectively identifies and filters false positives that may be generated by the first major model, ensuring that the final error list used for text correction has higher reliability and improving the precision and credibility of the correction results.

[0117] In addition, the second major model pre-building method in step S3 includes:

[0118] Collect sensitive word data and assign category labels to each sensitive word. ; and organized into a sensitive word database by category. and for sensitive word database Each sensitive word in the text is marked with a risk level. ;

[0119] Based on the second pre-trained large language model, the main parameters are frozen, and a sensitive word library embedding matrix is ​​injected into the input layer of the second pre-trained large language model. ;in, For sensitive word database The total number of sensitive words in the database, i.e., the sensitive word database The number of elements in the middle; The dimension of the embedded vector;

[0120] Based on the compiled sensitive word database Constructing a sensitive word training set and sensitive word verification set ;

[0121] The second pre-trained large model uses a configuration-based loss function. Training was conducted; among them, Training set for sensitive words The number of labeled samples in the data; Training set for sensitive words The Middle Sensitive word labels for each sample ; This is the inference function for the second pre-trained large model; This is the text corresponding to the correction text or error-free message; This is a vector concatenation operation;

[0122] In the sensitive word verification set The accuracy of sensitive word identification is calculated above. Recall rate ,satisfy and When that happens, save the current model as the second largest model.

[0123] It integrates sensitive word data from multiple channels, including: public standard libraries, industry-specific words, and trending online words.

[0124] Categorize by semantics or risk type, for each sensitive word Assign category labels The risk level is marked using a continuous value from 0 to 1. .

[0125] A general-purpose large language model (such as Llama-2 or Bloom) or a domain-specific model (such as FinBERT for financial text) is selected as the second pre-trained large language model, with a hidden layer dimension of [missing value]. (General model) or (Domain model).

[0126] Constructing a sensitive word library embedding matrix (where n is the total number of sensitive words), generated in the following way:

[0127] For each sensitive word Generate vectors using the word embedding layer of the base model. ;

[0128] vector spliced ​​into a matrix Injected into the input layer of the second pre-trained large language model (by modifying the model's input encoding logic to make the text vectors and...) Parallel computation is involved.

[0129] sensitive word database The vocabulary in the sample is paired with real text (violation text containing sensitive words and compliance text without sensitive words) to generate labeled samples:

[0130] Positive samples (containing sensitive words): ( () For texts containing sensitive words, such as "Buying ×× medicine can cure cancer 100%");

[0131] Negative samples (excluding sensitive words): ( () (For compliant text).

[0132] Divided into sensitive word training sets in a 7:3 ratio and sensitive word verification set Ensure that the training set covers all category labels and that the validation set contains typical sensitive word variants (such as homophones and split words).

[0133] Freeze 90% of the main parameters of the base model, and only fine-tune the embedding layer and the top classifier head, with a learning rate of [missing information]. .

[0134] Furthermore, the method for generating sensitive word suggestions in step S3 includes:

[0135] The second model detects sensitive words. Extract its position in the correction text or the text corresponding to the error-free message. Risk level Generate sensitive word suggestions ;

[0136] Sensitive word alerts Concatenate with the text context to calculate the association score. : ;in, The second largest model determines the probability that "sensitive words are context-dependent"; and when When this happens, it is identified as a "false positive sensitive word" and filtered.

[0137] The second model receives correction text or error-free message text. Then, the text content is traversed, and all words k (including variants, such as homophones and split words) that hit the sensitive word library K are marked by the pre-trained sensitive word recognition module.

[0138] Record the character index of sensitive words in the text (e.g., "violation content" appears in characters 10-16 of the text, position). ); calling the risk level pre-stored in the sensitive word library K. (such as "illegal content") ).

[0139] Generate sensitive word suggestions in JSON format. It contains the following fields:

[0140] Word: Original text containing sensitive words (e.g., "illegal content");

[0141] pos: Position index (e.g., [10,16]);

[0142] risk: Risk level (e.g., 0.8).

[0143] Extracting the parts before and after the sensitive words The first few characters are used as context (if insufficient, the first and last characters of the text are used), along with sensitive word prompts. Concatenate into a new text sequence (e.g., "...This is an explanation of the violation...").

[0144] Calling the probabilistic inference function of the second largest model Calculate the probability that "sensitive words are relevant to the context". Related scores Defined as: .

[0145] The preset correlation score threshold is 0.6 (general scenario), which can be adjusted to 0.65 for professional fields (such as legal texts).

[0146] when When a sensitive word is deemed a "false alarm" (e.g., "apple" appears in a fruit review text, which, although it hits the "apple" brand sensitive word database, is irrelevant to the context semantics), the alert is filtered out; when the correlation score is... When the value is greater than or equal to 0.6, retain the sensitive word prompt. .

[0147] This invention utilizes a pre-trained large language model to replace traditional rule-based or shallow machine learning models. Both the first and second large models possess powerful semantic understanding and context modeling capabilities. This enables the system to effectively handle the diversity and complexity of language, including emerging vocabulary, internet slang, technical terms, and texts containing long-distance dependencies and complex semantics, significantly improving the accuracy and generalization ability of text correction and sensitive word identification. Through the autonomous learning capability of the large model, the reliance on manually customized rules and dictionaries is greatly reduced, effectively overcoming the problems of high rule maintenance costs and poor adaptability in traditional methods.

[0148] Furthermore, in the sensitive word analysis stage, this invention calculates the correlation score between sensitive word prompts and context, and uses a threshold-based filtering mechanism to effectively identify and exclude "falsely reported sensitive words" that are harmless due to context, significantly reducing the false alarm rate of sensitive word identification and making the results more practically valuable.

[0149] In addition, step S4 includes:

[0150] The error list output by the first major model Sensitive word suggestions from the second largest model output Parsing JSON data into structured data; where, This indicates the location of the text error. For text error types; For the revised content;

[0151] Merge the parsed results in ascending order of text position coordinates to generate a unified result list: ;in, This is a list of the merged results; This is a set merging operation; This is a function for sorting by position.

[0152] It is clear that the outputs of the first major model (error list) and the second major model (sensitive word prompt) are both JSON files / strings. The JSON field specifications must be agreed upon in advance (e.g., the error list must contain pos_e (position), type_e (error type), and corr_e (correction content); the sensitive word prompt must contain pos_k (position), word (sensitive word text), and risk (risk level)).

[0153] Use the JSON viewer plugin in general text editing software (such as Notepad++) or the "Import from JSON" function in office software (such as Excel) to convert JSON data into a visual table / list, making it easier to manually verify the integrity of fields.

[0154] Error list mapping rules: pos_e → "location", type_e → "type", and corr_e → "content" are uniformly marked as "error" category; Sensitive word hint mapping rules: pos_k → "location", word → "content", and risk → "risk level" are uniformly marked as "sensitive word" category; For example, a row of data in the error list can be organized as: Location: [10,15], Type: Word error, Content: Correction content, Category: Error; Sensitive word hints can be organized as: Location: [20,25], Type: Sensitive word, Content: Sensitive word original text, Risk: 0.8, Category: Sensitive word.

[0155] Create a new Excel spreadsheet and paste the structured data for the error list and sensitive word prompts into different worksheets; then merge them into the same worksheet using "copy and paste" and perform a deduplication merge (manually filter entries with duplicates to retain more detailed information).

[0156] Extract the "starting coordinates" of each data point (e.g., [10,15] takes 10); sort by starting coordinates in ascending order, using Excel's "Sort" function (sort by the "starting coordinates" column in ascending order), and manually adjust any abnormal positions (e.g., positions spanning paragraphs need to be manually confirmed to determine if they belong to the same text stream).

[0157] The sorted and merged data is then organized into a structured list in a Word document according to the order of "location → type → content → additional information (such as risk level)" (e.g., 1. Location [10,15], type: word error, content: corrected content (error); 2. Location [20,25], type: sensitive word, content: original text of sensitive word (risk 0.8)...).

[0158] In some embodiments, the large-model-based text processing system 200 may include multiple functional modules composed of computer program segments. The computer programs for each program segment in the large-model-based text processing system 200 may be stored in the memory of a computer device and executed by at least one processor to perform (see details). Figure 1 (Description) Functionality for text processing based on large models.

[0159] In this embodiment, the large-model-based text processing system 200 can be divided into multiple functional modules according to the functions it performs, such as... Figure 2 As shown. The functional modules may include: a text preprocessing module 210, a text correction module 220, a sensitive word analysis module 230, and a result processing module 240. The module referred to in this invention is a series of computer program segments that can be executed by at least one processor and perform a fixed function, and are stored in memory. In this embodiment, the functions of each module will be described in detail in subsequent embodiments.

[0160] Specifically, the text preprocessing module is used to preprocess the input text, including text extraction, word segmentation, normalization, and paragraph segmentation. The text correction module is used to input the preprocessed text into the pre-built first language model, outputting an error list or a no-error message. When the first language model outputs an error list, the module detects the error list and corrects the preprocessed text based on the detected error list. The sensitive word analysis module is used to input the corrected text or the text corresponding to the no-error message into the pre-built second language model, outputting a sensitive word message or a no-sensitive word message. The result processing module is used to merge the output results of the pre-built first language model and the pre-built second language model, format the merged output, output a correction document, and clean up the correction document after outputting it.

[0161] Additionally, the text preprocessing module works as follows: Since user input may not be plain text but rather text uploaded via documents such as Word documents, the Markdown tool is first used to extract the text from the document. Then, word segmentation is performed, using a dictionary or algorithm to accurately divide the continuous character sequence into independent words or tags. Next, the normalization stage is performed, standardizing the segmented words, including unifying capitalization, correcting common spelling errors, converting numbers and symbols, removing redundant whitespace and irrelevant punctuation, and possibly performing stemming or lemma reconstruction to merge lexical variants. Finally, paragraph segmentation is performed, dividing the normalized text stream into logically complete paragraph units by identifying specific delimiters (such as consecutive line breaks and indentation) or combining semantic coherence analysis. The final output is clearly structured and uniformly formatted text content.

[0162] Text Correction Module: Based on a pre-trained large language model (DeepSeek model), the module achieves efficient text correction through targeted fine-tuning. First, leveraging the general language understanding capabilities gained from pre-training the DeepSeek model on massive text datasets, it further fine-tunes on an annotated correction dataset using P-Tuning v2 technology. This technique adds cue vectors before the input sequence and also inserts cue vectors into the model's intermediate layers. By jointly optimizing these cue vectors, the model better captures the features of the correction task, enabling it to deeply learn the semantic coherence, contextual dependencies, and error patterns of the text. In the application phase, the module employs a dual correction mechanism. The large model analyzes potential errors in the input text, providing error types, erroneous texts, and corrective results, organized into a JSON format, and outputting a formatted error list to accurately identify spelling, grammar, and collocation errors. Then, the large model performs a second check on this error list to eliminate inappropriate or overly strict errors, ensuring more reasonable and accurate text correction results. Simultaneously, the error list is provided to the large model so that it can temporarily correct the text. The corrected text is then input into the sensitive word analysis module for sensitive word analysis, achieving bidirectional parallel processing.

[0163] Sensitive Word Analysis Module: Based on the DeepSeek large-scale model, this module combines the efficient computational characteristics of Multi-Head Latent Attention (MLA) and Hybrid Expert Model (MoE) to achieve a deep understanding of text semantics. To further enhance the dynamic recognition capability of new sensitive words, this module employs P-Tuning v2 fine-tuning technology: by freezing the core parameters of the model and injecting only trainable continuous prompt vectors (soft prompts) at each layer, these prompts are deeply optimized using a bidirectional LSTM or MLP encoder, significantly enhancing the model's sensitivity to capturing sensitive word boundary variant expressions (such as homophonic abbreviations) while avoiding the overfitting risk of full-parameter fine-tuning. In the Retrieval Augmentation Generation (RAG) framework, the text to be examined first generates a 768-dimensional dense vector through the nomic-embedded-text-v1.5 model. This model, trained and optimized through multi-tasks, performs excellently in semantic similarity calculation and fine-grained classification. The sensitive word database employs a hybrid retrieval strategy (70% vector retrieval + 30% keyword matching): it leverages vector databases such as FaISS or Milvus to quickly recall semantically relevant sensitive word entries, then integrates traditional regular expressions to accurately intercept sensitive word variants, forming a dual-channel retrieval engine. The search results are injected into the large model as contextual knowledge, guiding DeepSeek to perform multiple rounds of inference in conjunction with the sensitive word database. An attention mechanism dynamically weights the relevance between the retrieved fragments and the original text, ultimately outputting sensitive word tags and location information. New sensitive words are collected through a combination of web crawling and manual review, and automatically categorized into the existing database using semantic similarity calculations by the large model, reducing manual maintenance costs.

[0164] The results processing module parses the results from the text correction and sensitive word analysis modules from a JSON list into plain text, merges the results, and then organizes them into paragraph styles that are easy for users to read. If the user inputs a document (such as a Word document), the python-docx library is used to parse the paragraph format of the original document, and a method of traversing and matching the error list is used to locate the error location and correct it in place. Finally, a corrected document that maintains the original document format is output, realizing automated document correction. To protect user privacy and avoid excessive document backlog, a scheduled task is used to periodically clean up the corrected documents generated on the server. The pseudocode for the results processing process is shown below:

[0165] # Input

[0166] input_type = "text" or "word_doc" # User input type

[0167] raw_content = path to the original text or Word document

[0168] correction_json = List of text correction JSON results

[0169] sensitive_json = List of JSON results from sensitive word analysis

[0170] # Processing flow

[0171] 1. Merge two JSON result sets → merged_errors (sorted by text position)

[0172] 2. IF input_type == "text":

[0173] Corrected text = Apply the modifications in merged_errors to the original text one by one.

[0174] Output = Formatted and corrected text into readable paragraphs

[0175] ELSE IF input_type == "word_doc":

[0176] Load Word document

[0177] FOR each paragraph in the document:

[0178] Locating the error location in the paragraph

[0179] Perform in-situ replacement (preserving the original formatting).

[0180] Output = Save the modified Word document.

[0181] Figure 3 This is a schematic diagram of the structure of a terminal 300 provided in an embodiment of the present invention. The terminal 300 can be used to execute the text processing method based on a large model provided in the embodiment of the present invention.

[0182] The terminal 300 may include a processor 310, a memory 320, and a communication module 330. These components communicate via one or more buses. Those skilled in the art will understand that the server structure shown in the figure does not constitute a limitation of the present invention. It may be a bus topology or a star topology, and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0183] The memory 320 can be used to store the execution instructions of the processor 310. The memory 320 can be implemented by any type of volatile or non-volatile memory terminal or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. When the execution instructions in the memory 320 are executed by the processor 310, the terminal 300 is able to perform some or all of the steps in the above method embodiments.

[0184] The processor 310 serves as the control center of the storage terminal, connecting various parts of the electronic terminal via various interfaces and lines. It executes software programs and / or modules stored in the memory 320, and calls data stored in the memory to perform various functions of the electronic terminal and / or process data. The processor can be composed of integrated circuits (ICs), such as a single packaged IC or multiple packaged ICs with the same or different functions connected together. For example, the processor 310 may consist only of a central processing unit (CPU). In this embodiment of the invention, the CPU may have a single processing core or include multiple processing cores.

[0185] The communication module 330 is used to establish a communication channel, enabling the storage terminal to communicate with other terminals. It receives user data sent by other terminals or sends user data to other terminals.

[0186] The present invention also provides a computer storage medium, wherein the computer storage medium may store a program, which, when executed, may include some or all of the steps provided in the embodiments of the present invention. The storage medium may be a magnetic disk, an optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0187] Those skilled in the art will clearly understand that the techniques in the embodiments of the present invention can be implemented using software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solutions in the embodiments of the present invention, or the parts that contribute to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium such as a USB flash drive, mobile hard drive, read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk, or other media capable of storing program code. It includes several instructions to cause a computer terminal (which may be a personal computer, server, or a second terminal, network terminal, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention.

[0188] The same or similar parts between the various embodiments in this specification can be referred to mutually. In particular, the terminal embodiments are basically similar to the method embodiments, so the description is relatively simple, and the relevant parts can be referred to the description in the method embodiments.

[0189] In the embodiments provided by this invention, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative. For instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between systems or modules may be electrical, mechanical, or other forms.

[0190] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0191] In addition, the functional modules in the various embodiments of the present invention can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module.

[0192] Although the present invention has been described in detail with reference to the accompanying drawings and preferred embodiments, the present invention is not limited thereto. Various equivalent modifications or substitutions can be made to the embodiments of the present invention by those skilled in the art without departing from the spirit and essence of the invention, and such modifications or substitutions should all be within the scope of the present invention. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should also be covered within the protection scope of the present invention.

Claims

1. A text processing method based on a large model, characterized in that, include: Step S1: Preprocess the input text, which includes extracting text, word segmentation, normalization, and paragraph segmentation in sequence. Step S2: Input the preprocessed text into the pre-built first model and output an error list or a no-error message; And when the first model outputs an error list, the output error list is detected, and the preprocessed text is corrected based on the detected error list; Step S3: Input the corrected text or the text corresponding to the error-free prompt into the pre-built second language model, and output sensitive word prompts or no sensitive word prompts; Step S4: Merge the output results of the first pre-built large model and the output results of the second pre-built large language model; format the merged output results, output the correction document, and clean up the correction document after outputting the correction document; The methods for pre-building the first major model in step S2 include: Historical text error correction annotation data is collected and organized into triples containing the original text, error location, and correction content. An error correction training set is then constructed based on the organized triples. and error correction verification set ; Based on the first pre-trained large model, the main parameters are frozen, and training cue vectors are inserted into the input layer of the first pre-trained large model. ,in, To indicate the length, The hidden layer dimension of the first pre-trained large model; The first pre-trained large model uses a configuration-based loss function. Training is performed, where N is the error correction training set. The number of samples in For the first The incorrect label of a sample This is the inference function of the first pre-trained large model. For input text, This is a vector concatenation operation; In the error correction verification set Calculate the error correction accuracy Recall rate ,satisfy and When the current model is saved as the first major model, save it as the first major model. The method for detecting the output error list in step S2 includes: According to the formula Calculate the consistency score of the error list ;in, For multiple rounds of reasoning; The initial error list output for the first large model; This is a variant of the error list for the j-th inference; For pre-stored text semantic similarity functions; The consistency score of the calculated error list With the preset self-consistency score threshold Compare; like If a false alarm is detected in the error list, it will be filtered out, and only the false alarm will be retained. Error entries, update the error list to .

2. The text processing method based on a large model according to claim 1, characterized in that, Step S1 includes: Determine if the input text is not a plain text document; if so, call the document parsing interface of the pre-stored MarkitDown tool to extract the text content; otherwise, directly read the text content of the input text; non-plain text documents include PDF documents and Word documents. The extracted text content is segmented by using a pre-stored word segmentation algorithm and a pre-defined dictionary. The segmented text content is standardized, including unifying capitalization, correcting spelling errors, converting number symbols, removing redundant whitespace and punctuation, stemming, and restoring word forms. The text content after normalization is processed to identify line breaks and encode adjacent sentences, and the semantic relevance of adjacent sentences is calculated; when the semantic relevance of adjacent sentences is lower than a set threshold, it is divided into a new paragraph. The text content after being divided into paragraphs is formed into preprocessed text.

3. The text processing method based on a large model according to claim 1, characterized in that, The second major model pre-building method in step S3 includes: Collect sensitive word data and assign category labels to each sensitive word. ; and organized into a sensitive word database by category. and for sensitive word database Each sensitive word in the text is marked with a risk level. ; Based on the second pre-trained large language model, the main parameters are frozen, and a sensitive word library embedding matrix is ​​injected into the input layer of the second pre-trained large language model. ;in, For sensitive word database The total number of sensitive words in the database, i.e., the sensitive word database The number of elements in the middle; The dimension of the embedded vector; Based on the compiled sensitive word database Constructing a sensitive word training set and sensitive word verification set ; The second pre-trained large model uses a configuration-based loss function. Training was conducted; among them, Training set for sensitive words The number of labeled samples in the data; Training set for sensitive words The Middle Sensitive word labels for each sample ; This is the inference function for the second pre-trained large model; This is the text corresponding to the correction text or error-free message; This is a vector concatenation operation; In the sensitive word verification set The accuracy of sensitive word identification is calculated above. Recall rate ,satisfy and When that happens, save the current model as the second largest model.

4. The text processing method based on a large model according to claim 3, characterized in that, The method for generating sensitive word suggestions in step S3 includes: The second model detects sensitive words. Extract its position in the correction text or the text corresponding to the error-free message. Risk level Generate sensitive word suggestions ; Sensitive word alerts Concatenate with the text context to calculate the association score. : ;in, This is the probabilistic inference function for the second largest model; The second largest model determines the probability that "sensitive words are context-dependent"; and when When this happens, it is identified as a "false positive for sensitive words" and filtered out.

5. The text processing method based on a large model according to claim 4, characterized in that, Step S4 includes: The error list output by the first major model Sensitive word suggestions from the second largest model output Parsing JSON data into structured data; where, This indicates the location of the text error. For text error types; For the revised content; Merge the parsed results in ascending order of text position coordinates to generate a unified result list: ;in, This is a list of the merged results; This is a set merging operation; This is a function for sorting by position.

6. A text processing system based on a large model, characterized in that, The text processing system comprising the large-model-based text processing method according to any one of claims 1-5 includes: The text preprocessing module is used to preprocess the input text. The preprocessing includes text extraction, word segmentation, normalization, and paragraph segmentation, which are performed sequentially. The text correction module is used to input the pre-processed text into the pre-built first model, output an error list or an error-free message; and when the first model outputs an error list, it detects the output error list and corrects the pre-processed text based on the detected error list. The sensitive word analysis module is used to input the corrected text or the text corresponding to the error-free prompt into the pre-built second language model and output sensitive word prompts or no sensitive word prompts. The results processing module is used to merge the output results of the pre-built first large model and the output results of the pre-built second large language model; format the merged output results; output a correction document; and clean up the correction document after outputting the correction document.

7. A terminal, characterized in that, include: processor; Memory used to store the processor's execution instructions; The processor is configured to perform the method according to any one of claims 1-5.

8. A computer-readable storage medium storing a computer program, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-5.

Images