Text processing method and system based on large model, terminal and storage medium

Abstract

The invention belongs to the technical field of text processing, and particularly relates to a text processing method and system based on a large model, a terminal and a storage medium, and the method comprises the steps: preprocessing an input text, inputting the preprocessed text into a pre-established first large model, and outputting an error list or an error-free prompt; when the first large model outputs an error list, the output error list is detected, and the preprocessed text is corrected according to the detected error list; and inputting the corrected text or the text corresponding to the error-free prompt into a pre-established second large language model, and outputting a sensitive word prompt or a sensitive word-free prompt. According to the method, the sensitive word problem possibly introduced or triggered by the error correction operation is actively recognized and processed through the serial flow of error correction and sensitive word analysis. And the second model performs sensitive word analysis on the basis of the error-corrected text, so that the real-time accuracy and safety of sensitive word recognition are remarkably improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of text processing, and in particular relates to a text processing method, system, terminal and storage medium based on a large model. Background Art

[0002] In today's digital age, the demand for text data processing is growing, especially in the areas of text error correction and sensitive word analysis. Traditional methods have many limitations when addressing these issues. For example, rule-based methods rely on manually formulated grammatical rules and dictionaries, making them difficult to adapt to the diversity and complexity of languages. This is especially true when dealing with emerging vocabulary and internet buzzwords, as rule maintenance costs are extremely high. Traditional machine learning methods, such as hidden Markov models (HMMs) and conditional random fields (CRFs), while capable of processing some simple text features, perform poorly when dealing with long-range dependencies and complex semantics, and their generalization capabilities for specialized terminology and emerging vocabulary are poor.

[0003] Furthermore, traditional methods are inefficient when processing large-scale text streams, making them difficult to meet real-time requirements. More importantly, existing technologies typically run text correction and sensitive word analysis as separate modules, ignoring the potential for sensitive words to appear in the text after correction, resulting in inaccurate sensitive word identification. For example, certain words may acquire new meanings after correction, becoming sensitive words, and traditional methods are unable to effectively identify such changes. Summary of the Invention

[0004] In view of the defect in the prior art that text correction and sensitive word analysis are usually run as independent modules, ignoring the sensitive word problems that may appear in the text after error correction, resulting in insufficient accuracy in sensitive word recognition, the present invention provides a text processing method, system, terminal and storage medium based on a large model to solve the above technical problems.

[0005] In a first aspect, the present invention provides a text processing method based on a large model, comprising: Step S1: preprocessing the input text, which includes sequentially performing text extraction, word segmentation, normalization, and paragraph segmentation operations; Step S2: inputting the preprocessed text into a pre-built first large model, outputting an error list or a clear error prompt; and when the first large model outputs an error list, detecting the output error list, and correcting the preprocessed text based on the detected error list; Step S3: input the corrected text or the text corresponding to the error-free prompt into the pre-built second language model, and output a sensitive word prompt or a no-sensitive word prompt; Step S4: Merge the output result of the pre-built first model and the output result of the pre-built second language model; format the merged output result, output the correction document at the same time, and clean up the correction document after output.

[0006] A further improvement of this technical solution is that step S1 includes: Determine whether the input text is a non-plain text document; if so, call the document parsing interface of the pre-existing MarkitDown tool to extract the text content; if not, directly read the text content of the input text; non-plain text documents include PDF documents and Word documents; Use the pre-existing word segmentation algorithm to match and segment the extracted text content according to the preset dictionary; Normalize the segmented text content, including capitalization, spelling correction, number conversion, removal of redundant spaces and punctuation, stemming, and word form restoration. After normalization, the text content is identified by line breaks and adjacent sentences are encoded, and the semantic relevance of adjacent sentences is calculated. When the semantic relevance of adjacent sentences is lower than the set threshold, the text is divided into a new paragraph. The text content after dividing the paragraphs is formed into preprocessed text.

[0007] A further improvement of the technical solution is that the method for pre-building the first large model in step S2 includes: Collect historical text error correction annotation data, organize it into triples containing the original text, error location and correction content, and build an error correction training set based on the organized triples and error correction validation set ; Based on the first pre-trained large model, freeze the main parameters and insert the training hint vector into the input layer of the first pre-trained large model. ,in, is the prompt length, is the hidden layer dimension of the first pre-trained large model; The first pre-trained large model is based on the configuration loss function Training is performed, where N is the error correction training set The number of samples in For the The wrong labels of samples, is the inference function of the first pre-trained large model, To input text, It is a vector concatenation operation; In the error correction validation set Calculate the error correction accuracy , recall rate ,satisfy and , save the current model as the largest model.

[0008] A further improvement of the technical solution is that the method for detecting the output error list in step S2 includes: According to the formula Compute the self-consistency score of the error list ;in, is the number of rounds of reasoning; is the error list variant for the jth reasoning; It is a pre-stored text semantic similarity function; The self-consistency score of the calculated error list With the preset self-consistent score threshold Make comparisons; like , then it is determined that there are false positive errors in the error list and the false positive errors are filtered out, and the remaining Error items, update the error list to .

[0009] A further improvement of this technical solution is that the method for pre-building the second largest model in step S3 includes: Collect sensitive word data and assign category labels to each sensitive word ; and organized into sensitive word libraries by category , and sensitive word library Each sensitive word in the risk level ; Based on the second pre-trained large language model, freeze the main parameters and inject the sensitive vocabulary embedding matrix into the input layer of the second pre-trained large language model. ;in, Sensitive word library The total number of sensitive words in the sensitive word library The number of elements in is the dimension of the embedding vector; According to the sensitive word library Constructing a sensitive word training set and sensitive word verification set ; The second pre-trained large model is based on the configuration loss function Conduct training; among them, Sensitive word training set The number of labeled samples in ; Sensitive word training set Middle Sensitive word labels of samples, ; The inference function of the second pre-trained large model; Correct the text or provide the corresponding text for error-free prompts; It is a vector concatenation operation; In the sensitive word verification set Calculate the accuracy of sensitive word recognition , recall rate ,satisfy and , save the current model as the second largest model.

[0010] A further improvement of this technical solution is that the method for generating sensitive word prompts in step S3 includes: The second largest model detects sensitive words , extract its position in the correction text or the corresponding text of the error-free prompt , risk level , generate sensitive word prompts ; Prompt sensitive words Splice with text context and calculate relevance score : ;in, The second largest model determines the probability of "sensitive words are related to the context"; and when When it is detected, it is judged as a "false positive sensitive word" and filtered.

[0011] A further improvement of this technical solution is that step S4 includes: Output the error list of the first model And the sensitive word prompts output by the second largest model Parse from JSON format into structured data; The text is in the wrong position; For text error type; To amend the content; Merge the parsing results in ascending order of text position coordinates to generate a unified result list: ;in, is the merged result list; It is a set merge operation; is a position sorting function.

[0012] In a second aspect, the present invention provides a text processing system based on a large model, comprising: The text preprocessing module is used to preprocess the input text. The preprocessing includes text extraction, word segmentation, normalization and paragraph segmentation. a text error correction module, configured to input the preprocessed text into a pre-built first model and output an error list or a clear error prompt; and, when the first model outputs an error list, to detect the output error list and to correct the preprocessed text based on the detected error list; The sensitive word analysis module is used to input the corrected text or the text corresponding to the error-free prompt into the pre-built second language model and output a sensitive word prompt or a non-sensitive word prompt; The result processing module is used to merge the output results of the pre-built first model and the output results of the pre-built second language model; format the merged output results, output the correction document at the same time, and clean up the correction document after the correction document is output.

[0013] In a third aspect, the present invention provides a terminal, comprising: processor, memory, wherein The memory is used to store computer programs, The processor is used to call and run the computer program from the memory, so that the terminal executes the above-mentioned terminal method.

[0014] In a fourth aspect, the present invention provides a computer storage medium, wherein the computer-readable storage medium stores instructions, which, when executed on a computer, enable the computer to execute the methods described in the above aspects.

[0015] The beneficial effects of the present invention are: By replacing traditional rule-based or shallow machine learning models with pre-trained large language models, both the first and second large models possess powerful semantic understanding and contextual modeling capabilities. This enables the system to effectively handle the diversity and complexity of language, including emerging vocabulary, internet slang, professional terminology, and text containing long-range dependencies and complex semantics. This significantly improves the accuracy and generalization of text error correction and sensitive word identification. The large model's autonomous learning capabilities significantly reduce reliance on manually customized rules and dictionaries, effectively overcoming the high rule maintenance costs and poor adaptability of traditional methods.

[0016] Through a sequential process of error correction followed by sensitive word analysis, we proactively identify and address sensitive word issues that may be introduced or triggered by error correction. The second model performs sensitive word analysis based on the corrected text, ensuring that sensitive word identification is based on the most accurate textual content. This completely resolves the critical flaw of ignoring sensitive word issues that may appear in text after error correction, significantly improving the real-time accuracy and security of sensitive word identification.

[0017] During the text correction phase, a self-consistency check mechanism for the error list was added. This mechanism, through multiple inferences and semantic similarity comparisons, effectively identifies and filters out potential false positives from the first model, ensuring the ultimate reliability of the error list used for text correction and improving the precision and credibility of the correction results.

[0018] During the sensitive word analysis stage, by calculating the correlation score between sensitive word prompts and the context, and based on the threshold judgment filtering mechanism, it is possible to effectively identify and eliminate "false positive sensitive words" that have no actual harm due to the context, greatly reducing the false alarm rate of sensitive word identification and making the results more valuable for practical reference. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0020] Figure 1 A schematic flow chart of a method according to an embodiment of the present invention.

[0021] Figure 2 A schematic block diagram of a system according to an embodiment of the present invention.

[0022] Figure 3 A schematic diagram of the structure of a terminal provided by an embodiment of the present invention.

[0023] 210 is a text preprocessing module, 220 is a text error correction module, 230 is a sensitive word analysis module, and 240 is a result processing module. DETAILED DESCRIPTION

[0024] In order to make the purpose, features, and advantages of the present invention more obvious and easy to understand, the technical solutions of the present invention will be clearly and completely described below in conjunction with the drawings in the specific embodiments. Obviously, the embodiments described below are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0025] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used in this specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention.

[0026] Figure 1The present invention provides a schematic flow chart of a text processing method based on a large model. Figure 1 The execution subject can be a text processing system based on a large model. According to different requirements, the order of the steps in the flowchart can be changed, and some steps can be omitted.

[0027] like Figure 1 As shown, the method includes: Step S1: preprocessing the input text, which includes sequentially performing text extraction, word segmentation, normalization, and paragraph segmentation operations; Step S2: inputting the preprocessed text into a pre-built first large model, outputting an error list or a clear error prompt; and when the first large model outputs an error list, detecting the output error list, and correcting the preprocessed text based on the detected error list; Step S3: input the corrected text or the text corresponding to the error-free prompt into the pre-built second language model, and output a sensitive word prompt or a no-sensitive word prompt; Step S4: Merge the output result of the pre-built first model and the output result of the pre-built second language model; format the merged output result, output the correction document at the same time, and clean up the correction document after output.

[0028] This invention proactively identifies and addresses sensitive word issues that may be introduced or triggered by error correction through a sequential process of error correction followed by sensitive word analysis. The second model performs sensitive word analysis based on the corrected text, ensuring that sensitive word identification is based on the most accurate textual content. This fundamentally addresses the critical issue of ignoring sensitive word issues that may appear in post-correction text, significantly improving the real-time accuracy and security of sensitive word identification.

[0029] To facilitate understanding of the present invention, the following further describes the text processing method based on a large model provided by the present invention based on the principle of the text processing method based on a large model in combination with the process of processing text based on a large model in the embodiment.

[0030] Wherein, step S1 includes: Determine whether the input text is a non-plain text document; if so, call the document parsing interface of the pre-existing MarkitDown tool to extract the text content; if not, directly read the text content of the input text; non-plain text documents include PDF documents and Word documents; Use the pre-existing word segmentation algorithm to match and segment the extracted text content according to the preset dictionary; Normalize the segmented text content. The normalization operations include unifying case, correcting text spelling mistakes, converting number symbol formats, removing redundant whitespace and punctuation, stemming, and lemmatization; Identify line breaks in the normalized text content and encode adjacent sentences, calculating the semantic correlation degree of adjacent sentences; when the semantic correlation degree of adjacent sentences is lower than the set threshold, divide them into new paragraphs; Form a preprocessed text from the text content after paragraph division.

[0031] Judge the input text type by the file extension (such as.pdf,.docx,.txt). If the extension is.pdf or.docx, it is determined as a non-pure text document; if it is.txt, it is determined as a pure text document.

[0032] Call the parsing interfaces of pandoc (a cross-format document conversion tool) or Python libraries (PyPDF2 for PDF, python-docx for Word) to extract the text content. For example, use the document.paragraphs interface of python-docx to read the paragraphs of a Word document, and extract the text of PDF pages through the pages interface of PyPDF2.PdfReader, ensuring that the paragraph order and basic format (such as line breaks, list symbols) of the original text are retained.

[0033] For.txt files, directly read the text content through a file stream (such as the open() function in Python), avoiding redundant format parsing operations.

[0034] Adopt Jieba分词 (for Chinese) or NLTK's word_tokenize (for English) as the pre-stored word segmentation algorithm, supporting the processing of Chinese-English mixed texts.

[0035] Construct a preset dictionary containing common vocabulary (such as "artificial intelligence", "data processing"), domain terms (such as "limitation of action" in the legal field, "targeted therapy" in the medical field), and stop words (such as "的", "了", "and"). When segmenting words,优先匹配预设词典中的词汇,未匹配项按最大概率切分 (如“自然语言处理”按词典匹配为“自然语言”+“处理”,而非单字切分). (Note: There seems to be an incorrect Chinese expression in this sentence, it should be something like "优先 match the words in the preset dictionary first, and for the unmatched items, segment them with the highest probability (e.g., '自然语言处理' is segmented as '自然语言' + '处理' according to the dictionary instead of single characters).")

[0036] Split the extracted text content by sentence groups (ending with punctuation marks "。?!;"), and then perform word segmentation on each sentence group, outputting a list of word segmentation results (such as "我爱文本处理" → ["我", "爱", "文本处理"]).

[0037] Unify capitalization: Unify English words into lowercase (e.g. "TextProcessing" → "textprocessing"), while keeping Chinese words as they are (no case distinction).

[0038] Correcting text spelling errors: Call the pyenchant (English) or pinyin (Chinese) library to perform spelling verification on the word segmentation results. For example, "teh" is corrected to "the" and "xiang" is corrected to "xiang".

[0039] Convert number symbol formats: convert Roman numerals to Arabic numerals (e.g., "Ⅲ" → "3"), convert Chinese numerals to Arabic numerals (e.g., "50" → "50"), and unify date formats (e.g., "October 2023" → "2023-10").

[0040] Remove redundant spaces and punctuation: Delete consecutive spaces (keep single spaces) and meaningless line breaks, unify punctuation to half-width format (such as "," → ",",";" → ";"), and remove redundant punctuation at the beginning and end of the text (such as "

Text content

[0041] Stemming and lemmatization: Perform stemming on English words (e.g., "running" → "run") and lemmatization on Chinese words (e.g., "processed" → "processed").

[0042] Line break recognition: By detecting \n or \r\n marks in the text, the original paragraph boundaries are preliminarily identified.

[0043] Sentence Encoding and Semantic Relevance Calculation: The normalized sentence groups are converted into vectors (using Word2Vec or BERT embedding). The cosine similarity function Sim(·,·) is used to calculate the semantic relevance of adjacent sentences (value range [0,1]). For example, the sentence "The weather is fine today" has a high relevance (about 0.8) to "suitable for outdoor activities" and a low relevance (about 0.2) to "machine learning is a branch of artificial intelligence."

[0044] Paragraph segmentation threshold: Set the semantic relevance threshold to 0.3 (dynamically adjustable). When the relevance between adjacent sentences is ≥0.3, they are merged into the same paragraph; when the relevance is <0.3, they are split into new paragraphs. For example, "Meeting time: October 2023" and "Attendees: Zhang and Li" have a low relevance (<0.3) and are split into two paragraphs; "The meeting discussed the project progress" and "The next steps were determined" have a high relevance (>0.3) and are merged into one paragraph.

[0045] Integrate the text after paragraph division in the original order to form a preprocessed text with clear structure, in the format: [Text of Paragraph 1]\n[Text of Paragraph 2]\n... (\n is the paragraph separator), which is used as the input of the text error correction module.

[0046] In addition, the method for pre-building the first large model in step S2 includes: Collect historical text error correction annotation data, organize it into triples containing the original text, error location, and corrected content according to the format, and construct an error correction training set based on the content of the sorted triples and an error correction verification set ; Based on the first pre-trained large model, freeze the main parameters, and insert a training prompt vector into the input layer of the first pre-trained large model , where is the prompt length, is the hidden layer dimension of the first pre-trained large model; The first pre-trained large model is trained based on the configured loss function , where N is the number of samples in the error correction training set , is the th sample's error label, is the inference function of the first pre-trained large model, is the input text, is the vector concatenation operation; Calculate the error correction accuracy and recall rate on the error correction verification set . When and are satisfied, save the current model as the first large model.

[0047] Collect historical texts from multiple fields such as news reports, social media comments, official documents, etc., covering general scenarios and professional fields (such as medical, legal), to ensure data diversity.

[0048] Annotation rules: By combining manual annotation and machine pre-annotation, annotate the error location (accurate to character index), error type (such as spelling error, grammar error, semantic ambiguity, etc.) and corrected content in the original text, forming a triple format: (original text, list of error locations, list of corrected contents). For example, the triple for the original text "He will go to school tomorrow" is: ("He will go to school tomorrow", [(4, 5)], ["again"]) (changing "will" to "again", the error location is the 4th - 5th character).

[0049] Divide the triples into an error correction training set and an error correction verification set , where the sample size of the training set is no less than 100,000, and the validation set covers typical error types in various fields.

[0050] Use a large pre-trained model with strong semantic understanding capabilities as the basis, such as BERT-base (suitable for general scenarios) or MedicalBERT (suitable for the medical field), whose hidden layer dimension d is 768 (BERT-base).

[0051] Insert a trainable hint vector at the model input layer , where the hint length m is set to 10 (which can be adjusted dynamically based on the text length). The vector is initialized with normally distributed random values ​​and concatenated with the input text vector before being fed into the model. The hint vector guides the model to focus on the "error correction task." For example, the semantic embedding of "Please detect and correct errors in the following text: [text]" strengthens task orientation.

[0052] Freeze 90% of the main parameters of the base model (only fine-tune the top Transformer layer and the prompt vector), use the Adam optimizer, set the learning rate to 5e-5, the number of samples to 32, and the number of epochs to 10. After each epoch, evaluate the performance on the validation set.

[0053] In the error correction validation set Calculation: The error correction accuracy Acc is the number of errors corrected correctly divided by the total number of errors predicted by the largest model; the error correction recall Rec is the number of errors corrected correctly divided by the total number of errors actually present in the text. and When , stop training and save the current model parameters, and make the current model the first largest model.

[0054] The preprocessed input text (After word segmentation and normalization) input into the first large model, the first large model performs error prediction distribution ;in, For text The length of the text; The vocabulary size for the error type.

[0055] Setting the error probability threshold is 0.85 (can be adjusted dynamically according to the field), and the Lexical element: wrong position (records the index of the word in the text) and error type (Record the corresponding error type, such as "Grammatical Error - Misuse of Prepositions").

[0056] Furthermore, the method for detecting the output error list in step S2 includes: According to the formula Compute the self-consistency score of the error list ;in, is the number of rounds of reasoning; is the error list variant for the jth reasoning; It is a pre-stored text semantic similarity function; The self-consistency score of the calculated error list With the preset self-consistent score threshold Make comparisons; like , then it is determined that there are false positive errors in the error list and the false positive errors are filtered out, and the remaining Error items, update the error list to .

[0057] When the first large model outputs the initial error list After that, start the multi-round reasoning mechanism and set the number of reasoning (It can be adjusted dynamically according to the complexity of the text. ).

[0058] Generate different inference conditions by adjusting the input prompt template, for example: Tip 1: "Please check the text for grammatical errors, including inappropriate word usage and reversed word order"; Tip 2: "Focus on identifying semantic ambiguities in the text and ignore technical terminology"; Tip 3: "Check for errors that could cause misunderstandings from the reader's perspective" Each inference outputs an error list variant The variant content includes the error location, type and correction suggestions, and the format is consistent with the initial error list.

[0059] Calculate the initial error list and text semantic similarity function based on multiple rounds of variations Error list variants , calculate the self-consistency score of the error list The preset self-consistency score threshold is 0.7 (general purpose), which can be increased to 0.75 in professional fields (such as law and medicine), and optimized through validation set testing (ensuring that the self-consistency score of more than 90% of true errors is ≥ the threshold).

[0060] Score comparison logic: Calculate the self-consistent score of the initial error list After that, the preset self-consistent score threshold Make a comparison.

[0061] When the overall score is lower than the threshold, each error item in the initial error list is checked separately: the error item in the initial error list is counted. Number of occurrences in a variant (If an error occurs 3 times in 5 variants, ); Calculate the support rate of a single error item ,reserve Error items (i.e., appearing in at least 60% of the variants, considered true errors), filtered False positives (such as professional terms being misidentified as errors).

[0062] Combine the filtered error items into a new error list , the format is consistent with the initial list, and it is used as the error information that is finally output to the result processing module.

[0063] During the text correction phase, the present invention adds a self-consistency detection mechanism for the error list. This mechanism, through multiple inferences and semantic similarity comparisons, effectively identifies and filters out potential false positives from the primary model, ensuring the ultimate reliability of the error list used for text correction and improving the precision and credibility of the correction results.

[0064] In addition, the method for pre-building the second largest model in step S3 includes: Collect sensitive word data and assign category labels to each sensitive word ; and organized into sensitive word libraries by category , and sensitive word library Each sensitive word in the risk level ; Based on the second pre-trained large language model, freeze the main parameters and inject the sensitive vocabulary embedding matrix into the input layer of the second pre-trained large language model. ;in, Sensitive word library The total number of sensitive words in the sensitive word library The number of elements in is the dimension of the embedding vector; According to the sensitive word library Constructing a sensitive word training set and sensitive word verification set ; The second pre-trained large model is based on the configuration loss function Conduct training; among them, Sensitive word training set The number of labeled samples in ; Sensitive word training set Middle Sensitive word labels of samples, ; The inference function of the second pre-trained large model; Correct the text or provide the corresponding text for error-free prompts; It is a vector concatenation operation; In the sensitive word verification set Calculate the accuracy of sensitive word recognition , recall rate ,satisfy and , save the current model as the second largest model.

[0065] Integrate multi-channel sensitive word data, including: public standard library, industry-specific words, and new hot words on the Internet.

[0066] Classify by semantics or risk type, for each sensitive word Assigning category labels , and use continuous values ​​from 0 to 1 to mark the risk level .

[0067] Use a general large language model (such as Llama-2, Bloom) or a domain adaptation model (such as FinBERT for financial text) as the second pre-trained large language model, with a hidden layer dimension of (generic model) or (Domain Model).

[0068] Constructing sensitive word embedding matrix (n is the total number of sensitive words), generated in the following way: For each sensitive word , using the word embedding layer of the base model to generate vectors ; The vector Splicing into a matrix , inject into the input layer of the second pre-trained large language model (by modifying the model input encoding logic to make the text vector parallel participation in the calculation).

[0069] Sensitive word library Pair the vocabulary in with real text (illegal text containing sensitive words and compliant text without sensitive words) to generate labeled samples: Positive sample (containing sensitive words): ( )( Text containing sensitive words, such as "Buying ×× medicine can 100% cure cancer"); Negative samples (excluding sensitive words): ( )( for compliant text).

[0070] Divide into sensitive word training sets in a ratio of 7:3 and sensitive word verification set , ensure that the training set covers all category labels and the validation set contains typical sensitive word variants (such as homophones and split words).

[0071] Freeze 90% of the main parameters of the base model, and only fine-tune the embedding layer and the top classification head, learning rate .

[0072] Furthermore, the method for generating sensitive word prompts in step S3 includes: The second largest model detects sensitive words , extract its position in the correction text or the corresponding text of the error-free prompt , risk level , generate sensitive word prompts ; Prompt sensitive words Splice with text context and calculate relevance score : ;in, The second largest model determines the probability of "sensitive words are related to the context"; and when When it is detected, it is judged as a "false positive sensitive word" and filtered.

[0073] The second largest model receives the correction text or the corresponding text of the error-free prompt After that, the text content is traversed and all words k (including variants, such as homophones and split words) that hit the sensitive word library K are marked through the pre-trained sensitive word recognition module.

[0074] Record the character index of sensitive words in the text (for example, "illegal content" appears in the 10th to 16th characters of the text, position ); Call the risk level pre-stored in the sensitive word library K (For example, "illegal content" ).

[0075] Generate sensitive word prompts in JSON format , containing the fields: word: the original text of sensitive words (such as "illegal content"); pos: position index (such as [10,16]); risk: Risk level (e.g. 0.8).

[0076] Intercept sensitive words before and after characters as context (if insufficient, take the beginning and end of the text), and sensitive word prompts Splice into a new text sequence (e.g. "... This is a description of the offending content...").

[0077] Call the probability inference function of the second largest model , calculate the probability of "sensitive words and context related" , association score Defined as: .

[0078] The default relevance score threshold is 0.6 (general scenarios), and can be adjusted to 0.65 for professional fields (such as legal texts).

[0079] when When the sensitive word is judged as a "false positive" (for example, "apple" appears in the fruit review text, although it hits the "apple" brand sensitive word library, it is irrelevant to the context semantics), and the prompt is filtered; when the correlation score When it is greater than or equal to 0.6, sensitive word prompts are retained .

[0080] The present invention uses a pre-trained large language model to replace the traditional rule-based or shallow machine learning model. Both the first and second large models have powerful semantic understanding and context modeling capabilities. This enables the system to effectively handle the diversity and complexity of language, including emerging vocabulary, Internet slang, professional terms, and texts containing long-distance dependencies and complex semantics, significantly improving the accuracy and generalization capabilities of text error correction and sensitive word recognition. Through the autonomous learning ability of the large model, the reliance on manually customized rules and dictionary libraries is greatly reduced, thus effectively overcoming the problems of high rule maintenance costs and poor adaptability of traditional methods.

[0081] Moreover, during the sensitive word analysis stage, the present invention calculates the correlation score between the sensitive word prompt and the context, and uses a threshold-based filtering mechanism to effectively identify and exclude "false positive sensitive words" that have no actual harm due to the context, thereby significantly reducing the false positive rate of sensitive word recognition and making the results more valuable for practical reference.

[0082] In addition, step S4 includes: Output the error list of the first model And the sensitive word prompts output by the second largest model Parse from JSON format into structured data; The text is in the wrong position; For text error type; To amend the content; Merge the parsing results in ascending order of text position coordinates to generate a unified result list: ;in, is the merged result list; It is a set merge operation; is a position sorting function.

[0083] It is clear that the outputs of the first model (error list) and the second model (sensitive word prompts) are both JSON files / strings, and the JSON field specifications must be agreed in advance (for example, the error list must include pos_e (position), type_e (error type), corr_e (correction content); sensitive word prompts must include pos_k (position), word (sensitive word original text), risk (risk level)).

[0084] Use the JSON viewing plug-in of common text editing software (such as Notepad++) or office software (such as Excel's "Import from JSON" function) to convert JSON data into a visual table / list to facilitate manual verification of field integrity.

[0085] Error list mapping rules: pos_e→"position", type_e→"type", corr_e→"content" are uniformly marked as "error" category; sensitive word prompt mapping rules: pos_k→"position", word→"content", risk→"risk level" are uniformly marked as "sensitive word" category; for example, a row of data in the error list can be organized as: position: [10,15], type: word error, content: corrected content, category: error; sensitive word prompts can be organized as: position: [20,25], type: sensitive word, content: original text of sensitive word, risk: 0.8, category: sensitive word.

[0086] Create a new Excel spreadsheet and paste the structured data of the error list and sensitive word prompts into different worksheets respectively; merge them into the same worksheet through "copy-paste" and perform deduplication merging (manually filter out entries in duplicate positions to retain more detailed information).

[0087] Extract the "position starting coordinates" of each data item (e.g., [10,15], select 10); sort in ascending order by the starting coordinates, use Excel's "Sort" function (sort by the "position starting coordinates" column in ascending order), and manually adjust abnormal positions (e.g., positions that span paragraphs require manual confirmation to determine whether they belong to the same text flow).

[0088] The sorted merged data is organized into a structured list in Word documents in the order of "position → type → content → additional information (such as risk level)" (for example: 1. Position [10,15], type: word error, content: corrected content (error); 2. Position [20,25], type: sensitive word, content: original text of sensitive word (risk 0.8)...).

[0089] In some embodiments, the large model-based text processing system 200 may include multiple functional modules composed of computer program segments. The computer program of each program segment in the large model-based text processing system 200 may be stored in a memory of a computer device and executed by at least one processor to perform (see Figure 1 Description) Functions for text processing based on large models.

[0090] In this embodiment, the text processing system 200 based on the large model can be divided into multiple functional modules according to the functions it performs, such as Figure 2 As shown. The functional modules may include: a text preprocessing module 210, a text error correction module 220, a sensitive word analysis module 230, and a result processing module 240. The module referred to in the present invention refers to a series of computer program segments that can be executed by at least one processor and can perform fixed functions, which are stored in the memory. In this embodiment, the functions of each module will be described in detail in subsequent embodiments.

[0091] Specifically, the text preprocessing module is used to preprocess the input text, and the preprocessing includes text extraction, word segmentation, normalization and paragraph segmentation operations performed in sequence; the text error correction module is used to input the preprocessed text into a pre-built first model, and output an error list or a no-error prompt; and when the first model outputs an error list, the output error list is detected, and the preprocessed text is corrected according to the detected error list; the sensitive word analysis module is used to input the corrected text or the text corresponding to the no-error prompt into a pre-built second language model, and output a sensitive word prompt or a no-sensitive word prompt; the result processing module is used to merge the output results of the pre-built first model and the output results of the pre-built second language model; and format the merged output results, output the corrected document at the same time, and clean up the corrected document after the corrected document is output.

[0092] In addition, the text preprocessing module: Since the user input may not be pure text, but may be entered by uploading a document such as Word, the MarkitDown tool is first used to extract the text from the document, and then a word segmentation operation is performed, using a dictionary or algorithm to accurately segment the continuous character sequence into independent words or tokens. Then, the normalization phase begins, and the segmented words are standardized, including unifying capitalization, correcting common spelling errors, converting numeric symbols, removing redundant spaces and irrelevant punctuation, and possibly performing stem extraction or morphological restoration to merge vocabulary variants. Finally, paragraph segmentation is performed, and the normalized text stream is divided into logically complete paragraph units by identifying specific delimiters (such as consecutive line breaks and indentations) or combining semantic coherence analysis. The final output is text content with a clear structure and uniform format.

[0093] Text Correction Module: Based on a pre-trained large language model (the DeepSeek model), the module achieves efficient text correction through targeted fine-tuning. First, leveraging the general language understanding capabilities of the DeepSeek model, pre-trained on massive amounts of text data, the module is further fine-tuned on annotated error correction datasets using P-Tuning v2 technology. This technology prepends hint vectors to the input sequence and also inserts hint vectors into the model's intermediate layers. By jointly optimizing these hint vectors, the model better captures the characteristics of the error correction task, enabling it to deeply learn the semantic coherence, contextual dependencies, and error patterns of the text. During the application phase, the module employs a dual error correction mechanism, where the large model analyzes potential errors in the input text, generates the error type, error text, and correction results, and outputs a formatted error list in JSON format, accurately identifying spelling, grammar, and collocation errors. The large model then performs a secondary check on this error list to eliminate inappropriate or overly critical errors, ensuring more reasonable and accurate text correction results. At the same time, the error list is provided to the big model so that the big model can make temporary corrections to the text. The corrected text is input into the sensitive word analysis module to perform sensitive word analysis on the corrected text, realizing bidirectional parallel processing.

[0094] Sensitive Word Analysis Module: Built on the DeepSeek large-scale model, the module combines the computational efficiency of the Multi-Head Latent Attention (MLA) mechanism with the Mixture of Experts (MoE) model to achieve a deep understanding of text semantics. To further enhance the dynamic recognition of new sensitive words, the module employs P-Tuning v2 fine-tuning technology. This technique freezes the model's core parameters and injects only trainable continuous prompt vectors (soft prompts) at each layer. These prompts are then deeply optimized using a bidirectional LSTM or MLP encoder. This significantly enhances the model's sensitivity to sensitive word boundary variants (such as homophonic abbreviations) while avoiding the overfitting risk of full parameter fine-tuning. Within the Retrieval-Augmented Generation (RAG) framework, the retrieval text is first processed through the nomic-embed-text-v1.5 model to generate a 768-dimensional dense embedding. This model, optimized through multi-task training, excels in semantic similarity calculation and fine-grained classification. The sensitive word database utilizes a hybrid search strategy (70% vector search + 30% keyword matching): leveraging vector databases such as FaISS or Milvus to rapidly recall semantically related sensitive word entries, this is then integrated with traditional regular expressions to precisely intercept sensitive word variants, creating a dual-channel search engine. The search results are then injected into the large model as contextual knowledge, guiding DeepSeek to perform multiple rounds of reasoning against the sensitive word database. Using an attention mechanism, the relevance of the searched fragments to the original text is dynamically weighted, ultimately outputting sensitive word labels and location information. New sensitive words are collected from the sensitive word database through a combination of web crawlers and manual review. The large model's semantic similarity calculation automatically categorizes them into the existing database, reducing manual maintenance costs.

[0095] Result Processing Module: The results of the text error correction module and the sensitive word analysis module are first parsed from a JSON-formatted list into plain text, the results are merged, and then organized into a paragraph style that is easy for users to read. If the user inputs a document (such as a Word document), the python-docx library is used to parse the paragraph format of the original document. The error list is traversed to locate the error location and perform in-situ modification. Finally, the corrected document is output in the original document format, realizing automated document correction. To protect user privacy and avoid excessive document backlogs, a scheduled task is used to regularly clean up the corrected documents generated on the server. The pseudo code of the result processing process is as follows: # Input input_type = "text" or "word_doc" # User input type raw_content = raw text or Word document path correction_json = Text correction JSON result list sensitive_json = Sensitive word analysis JSON result list # Processing Flow 1. Merge two JSON result sets → merged_errors (sorted by text position) 2. IF input_type == "text": CorrectedText = Apply the modifications in merged_errors one by one to the original text Output = Formatted corrected text into readable paragraphs ELSE IF input_type == "word_doc": Load a Word document FOR each paragraph IN document: Locate the error in the paragraph Perform an in-place replacement (preserve original formatting) Output = Save the modified Word document.

[0096] Figure 3 This is a structural diagram of a terminal 300 provided in an embodiment of the present invention. The terminal 300 can be used to execute the large model-based text processing method provided in an embodiment of the present invention.

[0097] The terminal 300 may include a processor 310, a memory 320, and a communication module 330. These components communicate via one or more buses. Those skilled in the art will appreciate that the server structure shown in the figure does not limit the present invention. The server structure may be a bus structure or a star structure, and may include more or fewer components than shown, or may combine certain components or arrange the components differently.

[0098] Memory 320 can be used to store execution instructions of processor 310. Memory 320 can be implemented by any type of volatile or non-volatile storage device, or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk, or optical disk. When the execution instructions in memory 320 are executed by processor 310, terminal 300 can perform some or all of the steps in the above-described method embodiments.

[0099] The processor 310 is the control center of the storage terminal. It uses various interfaces and lines to connect various parts of the entire electronic terminal. It executes various functions of the electronic terminal and / or processes data by running or executing software programs and / or modules stored in the memory 320, and calling data stored in the memory. The processor can be composed of an integrated circuit (IC), for example, it can be composed of a single packaged IC, or it can be composed of multiple packaged ICs with the same or different functions. For example, the processor 310 can only include a central processing unit (CPU). In the embodiment of the present invention, the CPU can be a single computing core or multiple computing cores.

[0100] The communication module 330 is used to establish a communication channel so that the storage terminal can communicate with other terminals, receive user data sent by other terminals, or send user data to other terminals.

[0101] The present invention also provides a computer storage medium, wherein the computer storage medium may store a program that, when executed, may include some or all of the steps of each embodiment provided herein. The storage medium may be a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0102] Those skilled in the art will clearly understand that the techniques in the embodiments of the present invention can be implemented using software plus a necessary general-purpose hardware platform. Based on this understanding, the technical solutions in the embodiments of the present invention, or the portion that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium such as a USB flash drive, a mobile hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, among other media capable of storing program code, and includes instructions for causing a computer terminal (which can be a personal computer, a server, or a second terminal, a network terminal, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention.

[0103] In this specification, the same or similar parts between the various embodiments can be referred to each other. In particular, for the terminal embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the description in the method embodiment.

[0104] In the several embodiments provided by the present invention, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative. For example, the division of the modules is merely a logical function division. In actual implementation, there may be other division methods, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of systems or modules, and can be electrical, mechanical or other forms.

[0105] The modules described as separate components may or may not be physically separate, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules may be selected to achieve the purpose of the present embodiment according to actual needs.

[0106] In addition, each functional module in each embodiment of the present invention may be integrated into one processing module, or each module may exist physically separately, or two or more modules may be integrated into one module.

[0107] Although the present invention has been described in detail with reference to the accompanying drawings and in conjunction with preferred embodiments, the present invention is not limited thereto. Without departing from the spirit and essence of the present invention, persons of ordinary skill in the art may make various equivalent modifications or substitutions to the embodiments of the present invention, and such modifications or substitutions shall be within the scope of the present invention. Any changes or substitutions that can be easily conceived by persons skilled in the art within the technical scope disclosed in the present invention shall be within the scope of protection of the present invention.

Claims

1. A text processing method based on a large model, characterized in that: include: Step S1: preprocessing the input text, which includes sequentially performing text extraction, word segmentation, normalization, and paragraph segmentation operations; Step S2: input the pre-processed text into the pre-built first model, and output an error list or a no-error prompt; When the first model outputs an error list, the output error list is detected, and the preprocessed text is corrected according to the detected error list; Step S3: input the corrected text or the text corresponding to the error-free prompt into the pre-built second language model, and output a sensitive word prompt or a no-sensitive word prompt; Step S4: Merge the output result of the pre-built first model and the output result of the pre-built second language model; format the merged output result, output the correction document at the same time, and clean up the correction document after output.

2. The text processing method based on a large model according to claim 1, characterized in that: Step S1 includes: Determine whether the input text is a non-plain text document; if so, call the document parsing interface of the pre-existing MarkitDown tool to extract the text content; if not, directly read the text content of the input text; non-plain text documents include PDF documents and Word documents; Use the pre-existing word segmentation algorithm to match and segment the extracted text content according to the preset dictionary; Normalize the segmented text content, including capitalization, spelling correction, number conversion, removal of redundant spaces and punctuation, stemming, and word form restoration. After normalization, the text content is identified by line breaks and adjacent sentences are encoded, and the semantic relevance of adjacent sentences is calculated. When the semantic relevance of adjacent sentences is lower than the set threshold, the text is divided into a new paragraph. The text content after dividing the paragraphs is formed into preprocessed text.

3. The text processing method based on a large model according to claim 1, characterized in that: The method for pre-building the first large model in step S2 includes: Collect historical text error correction annotation data, organize it into triples containing the original text, error location and correction content, and build an error correction training set based on the organized triples and error correction validation set ; Based on the first pre-trained large model, freeze the main parameters and insert the training hint vector into the input layer of the first pre-trained large model. ,in, is the prompt length, is the hidden layer dimension of the first pre-trained large model; The first pre-trained large model is based on the configuration loss function Training is performed, where N is the error correction training set The number of samples in For the The wrong labels of samples, is the inference function of the first pre-trained large model, To input text, It is a vector concatenation operation; In the error correction validation set Calculate the error correction accuracy , recall rate ,satisfy and , save the current model as the largest model.

4. The text processing method based on a large model according to claim 3 is characterized in that: The method for detecting the output error list in step S2 includes: According to the formula Compute the self-consistency score of the error list ;in, is the number of rounds of reasoning; is the error list variant of the jth reasoning; is a pre-stored text semantic similarity function; The self-consistency score of the calculated error list With the preset self-consistent score threshold Make comparisons; like , then it is determined that there are false positive errors in the error list and the false positive errors are filtered out, and the remaining Error items, update the error list to .

5. The text processing method based on a large model according to claim 3 is characterized in that: The method for pre-building the second largest model in step S3 includes: Collect sensitive word data and assign category labels to each sensitive word ; and organized into sensitive word libraries by category , and sensitive word library Each sensitive word in the risk level ; Based on the second pre-trained large language model, freeze the main parameters and inject the sensitive vocabulary embedding matrix into the input layer of the second pre-trained large language model. ;in, Sensitive word library The total number of sensitive words in the sensitive word library The number of elements in is the dimension of the embedding vector; According to the sensitive word library Constructing a sensitive word training set and sensitive word verification set ; The second pre-trained large model is based on the configuration loss function Conduct training; among them, Sensitive word training set The number of labeled samples in ; Sensitive word training set Middle Sensitive word labels of samples, ; The inference function of the second pre-trained large model; Correct the text or provide the corresponding text for error-free prompts; It is a vector concatenation operation; In the sensitive word verification set Calculate the accuracy of sensitive word recognition , recall rate ,satisfy and , save the current model as the second largest model.

6. The text processing method based on a large model according to claim 5, characterized in that: The method for generating sensitive word prompts in step S3 includes: The second largest model detects sensitive words , extract its position in the correction text or the corresponding text of the error-free prompt , risk level , generate sensitive word prompts ; Prompt sensitive words Splice with text context and calculate relevance score : ;in, The second largest model determines the probability of "sensitive words are related to the context"; and when When it is detected, it is judged as a "false positive sensitive word" and filtered.

7. The text processing method based on a large model according to claim 6, characterized in that: Step S4 includes: Output the error list of the first model And the sensitive word prompts output by the second largest model Parse from JSON format into structured data; The text is in the wrong position; For text error type; To amend the content; Merge the parsing results in ascending order of text position coordinates to generate a unified result list: ;in, is the merged result list; It is a set merge operation; is a position sorting function.

8. A text processing system based on a large model, characterized in that: include: The text preprocessing module is used to preprocess the input text. The preprocessing includes text extraction, word segmentation, normalization and paragraph segmentation. The text error correction module is used to input the pre-processed text into the pre-built first model and output an error list or a clear error prompt; When the first model outputs an error list, the output error list is detected, and the preprocessed text is corrected according to the detected error list; The sensitive word analysis module is used to input the corrected text or the text corresponding to the error-free prompt into the pre-built second language model and output a sensitive word prompt or a non-sensitive word prompt; The result processing module is used to merge the output results of the pre-built first model and the output results of the pre-built second language model; format the merged output results, output the correction document at the same time, and clean up the correction document after the correction document is output.

9. A terminal, characterized in that: include: processor; a memory for storing execution instructions of the processor; The processor is configured to execute the method according to any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images