Text data cleaning method and system based on multi-level large model serialization

Abstract

The invention discloses a text data cleaning method and system based on multi-level large model serialization, and the method comprises the following steps: obtaining to-be-cleaned text data, and carrying out the error examination and preliminary cleaning of the to-be-cleaned text data through an initial layer large model, and obtaining the preliminarily cleaned text data; performing semantic consistency maintenance on the preliminarily cleaned text data based on a semantic similarity evaluation mechanism; unknown error understanding and efficient cleaning are conducted on the maintained text through a middle layer large model, and text data after efficient cleaning are obtained; and inputting the to-be-cleaned text data and the efficiently-cleaned text data into the improvement layer large model to carry out text quality improvement and optimization so as to complete text data cleaning. According to the method, the serial combination of the multi-level small-parameter large-language model is innovatively adopted to replace the large-parameter large-language model, the data cleaning task is completed, and therefore a data cleaning scheme better fitting the production environment is achieved.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence and text processing technology, and specifically relates to a text data cleaning method and system based on multi-level large model serialization. Background Art

[0002] With the rapid development of artificial intelligence (AI), large-scale language models (LLMs) have become a core technology in natural language processing (NLP). These models, pre-trained on massive amounts of text data using deep learning and neural network techniques, possess powerful language understanding and generation capabilities, capable of capturing complex patterns and long-range dependencies in language. However, large LMs face numerous challenges in practical applications. First, these models typically have billions or even hundreds of billions of parameters, placing exponential demands on computing resources and storage space. This high resource consumption makes large models difficult to deploy and run in resource-constrained environments. Second, the complexity of large models leads to slow inference speeds, making them difficult to meet real-time processing requirements. The collaborative working model of multi-layered, small-parameter models replaces a single large model with multiple, small-parameter models, each responsible for different tasks and functions. This approach significantly reduces computing resource requirements while maintaining overall performance. Furthermore, this collaborative working model model improves scalability and flexibility, allowing flexible adjustments to the model's hierarchical structure and parameter configuration based on specific application scenarios, opening up new possibilities for the efficient deployment of large models in production environments. By rationally allocating computing resources and optimizing model structures, large models can be run efficiently on different hardware platforms. This model also provides ample room for further model optimization and innovation, enabling specialized optimization and improvement of different small-parameter models, thereby improving overall system performance and efficiency.

[0003] In today's digital age, the explosive growth of text data has become a significant phenomenon. This text data is not only massive in volume but also complex and diverse in content, covering a wide range of information from daily life to professional fields, providing a rich data resource for scientific research, business analysis, and policymaking. However, text data is also filled with a large amount of invalid content and noise, such as URLs and advertising information. This content not only consumes data storage space but also interferes with semantic analysis of the text, reducing data quality and usability. In addition to these obvious issues, text data also contains subtle but equally inconspicuous issues that can affect semantic interpretation, such as image captions and page numbers embedded in text. Furthermore, typos and grammatical errors, while seemingly minor, can have a significant impact on text understanding, especially in applications requiring extremely high accuracy, such as legal documents and medical literature. To address these challenges, text data cleaning has become a crucial foundational task in the field of natural language processing. Text data cleaning involves using a series of methods and steps to remove invalid content and noise, correct errors, and standardize the format, thereby improving the quality of text data and making it more suitable for subsequent analysis and processing. This process not only affects the quality and availability of data but also directly impacts the effectiveness and accuracy of natural language processing tasks. As text data continues to grow and become more complex, text data cleaning technology is also constantly evolving and improving. Researchers and practitioners are actively exploring various cleaning methods, from simple regular expression matching and string replacement to intelligent cleaning models based on machine learning and deep learning, continuously driving the advancement of text data cleaning technology. However, how to efficiently and accurately clean large-scale and diverse text data, and how to cope with emerging new problems and scenarios, remain issues that text data cleaning technology needs to continuously explore and address. Summary of the Invention

[0004] The present invention aims to solve the deficiencies of the prior art and provides the following solutions:

[0005] A text data cleaning method based on multi-level large model serialization includes the following steps:

[0006] Acquire text data to be cleaned, and perform error review and preliminary cleaning on the text data to be cleaned using the initial layer large model to obtain preliminary cleaned text data;

[0007] Maintaining semantic consistency of the text data after preliminary cleaning based on a semantic similarity evaluation mechanism;

[0008] The middle-layer large model is used to understand unknown errors in the maintained text and efficiently clean it, thus obtaining efficiently cleaned text data.

[0009] The text data to be cleaned and the text data after efficient cleaning are input into the improvement layer model to improve and optimize the text quality, thereby completing the text data cleaning.

[0010] Preferably, the method for obtaining the preliminarily cleaned text data includes:

[0011] Obtaining a first system prompt word based on the text data to be cleaned and the domain information, and obtaining a first user prompt word through a known error list provided by a user;

[0012] The initial layer large model performs error review on the text to be cleaned using the first system prompt word and the first user prompt word;

[0013] The initial layer large model performs preliminary cleaning on the text data after error review to obtain the preliminary cleaned text data.

[0014] Preferably, the method for maintaining semantic consistency includes:

[0015] Mapping the text data to be cleaned and the text data after preliminary cleaning to a high-dimensional semantic space respectively using semantic embedding technology to obtain a vector representation of the text to be cleaned and a vector representation of the text after preliminary cleaning;

[0016] The similarity between the text vector representation to be cleaned and the text vector after preliminary cleaning is calculated by cosine similarity:

[0017]

[0018] Where Score represents the similarity, A represents the text vector after preliminary cleaning, B represents the text vector to be cleaned, · represents the vector dot product, and || || represents the modulus of the vector;

[0019] Setting a semantic similarity threshold, and comparing the obtained similarity with the semantic similarity threshold;

[0020] If the obtained similarity is greater than the semantic similarity threshold, there is no need to perform semantic consistency maintenance;

[0021] If the obtained similarity is less than the semantic similarity threshold, the inference parameters of the initial layer large model are updated and re-cleaned until the iteration reaches a similarity greater than the semantic similarity threshold, completing the semantic consistency maintenance.

[0022] Preferably, the method for obtaining the efficiently cleaned text data includes:

[0023] Use COT thinking chain prompts and structured templates to build the second system prompt words and the second user prompt words;

[0024] The second system prompt word and the second user prompt word are combined to perform unknown error understanding and efficient cleaning on the maintained text to obtain the efficiently cleaned text data.

[0025] Preferably, the method for improving and optimizing text quality includes:

[0026] Analyzing the difference between the text data to be cleaned and the text data after efficient cleaning by using the improvement layer large model;

[0027] The text errors in the efficiently cleaned text data are identified and corrected by analyzing the differences, thereby completing the text data cleaning.

[0028] The present invention also provides a text data cleaning system based on a multi-level large model serialization, wherein the system applies any of the above-mentioned methods and comprises: a preliminary cleaning module, a semantic maintenance module, an efficient cleaning module and a perfect optimization module;

[0029] The preliminary cleaning module is used to obtain text data to be cleaned, and perform error review and preliminary cleaning on the text data to be cleaned through the initial layer large model to obtain preliminary cleaned text data;

[0030] The semantic maintenance module performs semantic consistency maintenance on the text data after preliminary cleaning based on a semantic similarity evaluation mechanism;

[0031] The efficient cleaning module uses the middle layer large model to understand unknown errors and efficiently clean the maintained text to obtain efficiently cleaned text data;

[0032] The improvement and optimization module is used to input the text data to be cleaned and the text data after efficient cleaning into the improvement layer model to improve and optimize the text quality, thereby completing text data cleaning.

[0033] Preferably, the workflow of the preliminary cleaning module includes:

[0034] Obtaining a first system prompt word based on the text data to be cleaned and the domain information, and obtaining a first user prompt word through a known error list provided by a user;

[0035] The initial layer large model performs error review on the text to be cleaned using the first system prompt word and the first user prompt word;

[0036] The initial layer large model performs preliminary cleaning on the text data after error review to obtain the preliminary cleaned text data.

[0037] Preferably, the workflow of the semantic maintenance module includes:

[0038] Mapping the text data to be cleaned and the text data after preliminary cleaning to a high-dimensional semantic space respectively using semantic embedding technology to obtain a vector representation of the text to be cleaned and a vector representation of the text after preliminary cleaning;

[0039] The similarity between the text vector representation to be cleaned and the text vector after preliminary cleaning is calculated by cosine similarity:

[0040]

[0041] Where Score represents the similarity, A represents the text vector after preliminary cleaning, B represents the text vector to be cleaned, · represents the vector dot product, and || || represents the modulus of the vector;

[0042] Setting a semantic similarity threshold, and comparing the obtained similarity with the semantic similarity threshold;

[0043] If the obtained similarity is greater than the semantic similarity threshold, there is no need to perform semantic consistency maintenance;

[0044] If the obtained similarity is less than the semantic similarity threshold, the inference parameters of the initial layer large model are updated and re-cleaned until the iteration reaches a similarity greater than the semantic similarity threshold, completing the semantic consistency maintenance.

[0045] Preferably, the workflow of the efficient cleaning module includes:

[0046] Use COT thinking chain prompts and structured templates to build the second system prompt words and the second user prompt words;

[0047] The second system prompt word and the second user prompt word are combined to perform unknown error understanding and efficient cleaning on the maintained text to obtain the efficiently cleaned text data.

[0048] Preferably, the workflow of the improvement and optimization module includes:

[0049] Analyzing the difference between the text data to be cleaned and the text data after efficient cleaning by using the improvement layer large model;

[0050] The text errors in the efficiently cleaned text data are identified and corrected by analyzing the differences, thereby completing the text data cleaning.

[0051] Compared with the prior art, the present invention has the following beneficial effects:

[0052] The present invention innovatively adopts a serial combination of multi-level small-parameter large language models to replace the large-parameter large language model to complete the data cleaning task, thereby realizing a data cleaning solution that is more suitable for the production environment. Before the task starts, the user inputs the text data to be cleaned and sets the relevant cleaning parameters and goals, which provides a clear guidance direction for the serial processing of the multi-level large model. The multi-level large models that are linked together process the text data in series in turn. The models at each level focus on their specific task goals, from basic cleaning to in-depth mining of unknown errors, and then to retaining the original text content and semantics to the greatest extent, making the text cleaning process more systematic and organized, gradually improving the text cleaning effect, and significantly improving the efficiency and quality of text cleaning.

[0053] During the cleaning process, the present invention specifically introduces a semantic consistency maintenance mechanism based on semantic similarity evaluation. After the cleaning of each level is completed, the system will evaluate the semantic similarity between the cleaned text and the original text. If the similarity is lower than the preset threshold, it means that too many details may have been lost or hallucination problems have occurred during the cleaning process. The system will automatically adjust the inference parameters and return to the cleaning model of the current level to re-clean. This mechanism effectively guarantees the semantic consistency of the cleaning results and ensures that the text does not lose important information during the cleaning process. During the entire task execution process, the system monitors the processing status and effects of the models at each level in real time, and can dynamically adjust the model parameters and processing strategies according to the characteristics of the text data and the cleaning requirements. This flexible adjustment mechanism enables the system to better adapt to different types of text data and diverse cleaning scenarios, and achieve efficient and accurate text data cleaning, which is highly practical and widely applicable. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] In order to more clearly illustrate the technical solution of the present invention, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0055] Figure 1 Schematic diagram of a method flow in an embodiment of the present invention. DETAILED DESCRIPTION

[0056] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0057] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0058] Example 1

[0059] In this embodiment, if Figure 1 As shown, a text data cleaning method based on a multi-level large model serialization includes the following steps:

[0060] S1. Obtain the text data to be cleaned, and perform error review and preliminary cleaning on the text data to be cleaned through the initial layer large model to obtain the text data after preliminary cleaning.

[0061] The method for obtaining preliminarily cleaned text data includes: obtaining a first system prompt word based on the text data to be cleaned and domain information, and obtaining a first user prompt word through a known error list provided by a user; an initial layer large model performs error review on the text to be cleaned through the first system prompt word and the first user prompt word; the initial layer large model performs preliminarily cleaning on the text data after the error review to obtain preliminarily cleaned text data.

[0062] In this embodiment, domain information is first integrated into system prompts to enable the large model to better understand and process text data. Simultaneously, a user-provided list of known errors is converted into user prompts, which clearly indicate the types of errors that need to be identified and cleaned, such as URL links and advertising information. The initial-level large model uses these prompts to guide its detailed review of the text data. System prompts enable the model to identify specialized terminology and domain characteristics in the text, allowing it to more accurately locate and address domain-related errors during the review process. User prompts help the model identify known errors in the text and guide it in performing the corresponding cleaning operations. By combining system prompts with user prompts, the large model can comprehensively identify and address known errors in the text, ensuring the accuracy and efficiency of the cleaning process. After completing the detailed review of known errors, the initial-level large model will perform efficient cleaning operations. Based on the guidance of the prompts with added thought chains, the model deletes, replaces, or formats the identified errors to optimize the text's format and semantic structure. This process not only improves the quality of text data, but also lays a solid foundation for subsequent multi-layer large model serial processing, making the entire text data cleaning process more efficient, accurate, and reliable. The syntax of the prompt word template of the initial layer large model can be described as shown in Table 1:

[0063] Table 1

[0064]

[0065] Before executing a task, the user needs to enter information, and the specific information can be increased or decreased according to actual needs.

[0066] S2. Maintain the semantic consistency of the text data after preliminary cleaning based on the semantic similarity evaluation mechanism.

[0067] The method for maintaining semantic consistency includes: using semantic embedding technology to map the text data to be cleaned and the text data after preliminary cleaning to a high-dimensional semantic space, respectively, to obtain a vector representation of the text to be cleaned and a vector representation of the text after preliminary cleaning; and calculating the similarity between the vector representation of the text to be cleaned and the vector of the text after preliminary cleaning by cosine similarity:

[0068]

[0069] Where Score represents similarity, A represents the text vector after preliminary cleaning, B represents the text vector to be cleaned, · represents the vector dot product, and || || represents the modulus of the vector. A semantic similarity threshold is set, and the obtained similarity is compared with the semantic similarity threshold. If the obtained similarity is greater than the semantic similarity threshold, no semantic consistency maintenance is required. If the obtained similarity is less than the semantic similarity threshold, the inference parameters of the initial layer large model are updated and re-cleaned until the iteration reaches a similarity greater than the semantic similarity threshold, completing the semantic consistency maintenance.

[0070] In this embodiment, the calculated similarity value is compared with a preset, adjustable semantic similarity threshold θ. The thresholds θ for different levels should be set differently based on the cleaning objectives and complexity of each level. The initial layer primarily removes obvious noise and has relatively low requirements for semantic similarity, so a threshold of θ1 = 0.8 can be set. The intermediate layer addresses subtle issues and has higher requirements for semantic preservation, so a threshold of θ2 = 0.9 can be set. The refinement layer aims to reconstruct the text while preserving semantics to the greatest extent possible, while also removing all potential errors, so a threshold of θ3 = 0.7 can be set. If the similarity falls below this threshold, it indicates that excessive detail may have been lost during the cleaning process, or that hallucinations may have occurred, resulting in a significant semantic deviation between the cleaned text and the original text. At this point, the system automatically updates inference parameters, such as adjusting the Top-k parameter to change the number of candidate tokens considered by the model when generating the cleaning results, or adjusting the Temperature parameter to influence the probability distribution of the model output, making it more concentrated or more dispersed. After the parameters are updated, the text data is re-entered into the cleaning model at the current level for a new round of cleaning. This process can be iterated repeatedly until the semantic similarity between the cleaned text and the original text reaches or exceeds the set threshold, thereby ensuring that the cleaning result maintains a high degree of semantic consistency with the original text while retaining text details.

[0071] S3. Use the middle-layer large model to understand unknown errors and efficiently clean the maintained text to obtain efficiently cleaned text data.

[0072] The method for obtaining efficiently cleaned text data includes: using COT thinking chain prompts and structured templates to construct second system prompt words and second user prompt words; combining the second system prompt words and the second user prompt words to understand unknown errors and efficiently clean the maintained text to obtain efficiently cleaned text data.

[0073] In this embodiment, after the initial-layer model completes the cleaning of all issues, the preliminarily cleaned text data is received by the intermediate-layer model, which performs a deep understanding and efficient cleaning of the text for unknown errors. This portion of the prompting project utilizes techniques such as COT (Conceptual Object Interpretation) and structured template prompting to achieve a deep understanding and efficient cleaning of the text for unknown errors. When constructing system prompts, relevant domain information should be incorporated, with an emphasis on the system's ability to meticulously examine and accurately clean unknown issues in the text that may affect semantic logic. The preliminarily cleaned text data is embedded in a structured form within the user prompts. Using these carefully designed prompts, the intermediate-layer model guides it to conduct a deep and refined review and optimized processing of unknown errors within the preliminarily cleaned text data. Simultaneously, the model also incorporates extensive domain knowledge and contextual information to achieve a deep understanding of the text at the semantic level, thereby more accurately determining which content may interfere with or mislead semantic logic. Through these prompts, the model can more accurately locate and process unknown errors, significantly improving the quality and efficiency of text cleaning. The grammar of the prompt template for the intermediate-layer model can be described as shown in Table 2:

[0074] Table 2

[0075]

[0076] S4. Input the text data to be cleaned and the text data after efficient cleaning into the improvement layer model to improve and optimize the text quality and complete the text data cleaning.

[0077] The method for improving and optimizing text quality includes: using a large model of the improvement layer to analyze the differences between the text data to be cleaned and the text data after efficient cleaning; identifying text errors in the text data after efficient cleaning through the analyzed differences and correcting them to complete the text data cleaning.

[0078] In this embodiment, in order to ensure the integrity of the text to the greatest extent possible and to effectively solve problems such as text deletion, modification, and inappropriate summarization that may be introduced by the large language model during the processing process, the cleaned text data of the intermediate layer will be transmitted to the large model of the improvement layer together with the original text. In this link, the carefully designed prompt words will integrate cutting-edge technologies such as COT thinking chain prompts and structured template prompts to guide the model to deeply analyze the differences between the intermediate layer text cleaning data and the original text. Through this meticulous comparative analysis, the model can accurately identify text errors that have not yet been cleaned, and at the same time, in the process of correcting errors, it retains the content and semantics of the original text to the greatest extent possible, ensuring that the accuracy and integrity of the text data are fully guaranteed. The prompt word template of the large model part of the improvement layer can be described in its grammar as shown in Table 3:

[0079] Table 3

[0080]

[0081] Example 2

[0082] In this embodiment, a text data cleaning system based on a multi-level large model series includes: a preliminary cleaning module, a semantic maintenance module, an efficient cleaning module and a perfect optimization module.

[0083] The preliminary cleaning module is used to obtain the text data to be cleaned, and perform error review and preliminary cleaning on the text data to be cleaned through the initial layer large model to obtain the text data after preliminary cleaning.

[0084] The workflow of the preliminary cleaning module includes: obtaining the first system prompt word based on the text data to be cleaned and the domain information, and obtaining the first user prompt word through the known error list provided by the user; the initial layer large model performs error review on the text to be cleaned through the first system prompt word and the first user prompt word; the initial layer large model performs preliminary cleaning on the text data after the error review to obtain the text data after preliminary cleaning.

[0085] The semantic maintenance module maintains the semantic consistency of the text data after preliminary cleaning based on the semantic similarity evaluation mechanism.

[0086] The workflow of the semantic maintenance module includes: using semantic embedding technology to map the text data to be cleaned and the text data after preliminary cleaning into a high-dimensional semantic space, respectively, to obtain the vector representation of the text to be cleaned and the vector representation of the text after preliminary cleaning; and calculating the similarity between the vector representation of the text to be cleaned and the vector representation of the text after preliminary cleaning through cosine similarity:

[0087]

[0088] Where Score represents similarity, A represents the text vector after preliminary cleaning, B represents the text vector to be cleaned, · represents the vector dot product, and || || represents the modulus of the vector. A semantic similarity threshold is set, and the obtained similarity is compared with the semantic similarity threshold. If the obtained similarity is greater than the semantic similarity threshold, no semantic consistency maintenance is required. If the obtained similarity is less than the semantic similarity threshold, the inference parameters of the initial layer large model are updated and re-cleaned until the iteration reaches a similarity greater than the semantic similarity threshold, completing the semantic consistency maintenance.

[0089] The efficient cleaning module uses the middle-layer large model to understand unknown errors and efficiently clean the maintained text to obtain efficiently cleaned text data.

[0090] The workflow of the efficient cleaning module includes: using COT thinking chain prompts and structured templates to construct the second system prompt words and the second user prompt words; combining the second system prompt words and the second user prompt words to understand unknown errors and efficiently clean the maintained text to obtain efficiently cleaned text data.

[0091] The improvement and optimization module is used to input the text data to be cleaned and the text data after efficient cleaning into the improvement layer model to improve and optimize the text quality and complete the text data cleaning.

[0092] The workflow of the improvement and optimization module includes: using the improvement layer large model to analyze the differences between the text data to be cleaned and the text data after efficient cleaning; identifying and correcting text errors in the text data after efficient cleaning through the analyzed differences, and completing text data cleaning.

[0093] Example 3

[0094] In this embodiment, a piece of news is used as an example to introduce the process of the present invention:

[0095] S1. Initial layer cleaning:

[0096] Field Name: News

[0097] Text to be cleaned: With the emergence of AI technologies such as ChatGPT, Doubao, Kimi, and DeepSeek, and amidst the current surge of digitalization, AI writing, with its rapid text generation, extensive knowledge integration, time-saving costs, and increased writing efficiency, is poised to surpass or even replace traditional document writing. (Report / Feedback 02:37 · Xinhua Photo Selection) In-depth research has found that despite significant advantages, AI writing cannot truly replace document writing from a theoretical, theoretical, and ethical perspective. (Image source: Screenshot of an online AI writing assistant search) AI writing and traditional document writing complement each other and operate in tandem. In the context of AI writing, professionals in organizations must not only discern and adapt to trends, but also proactively capitalize on them and avoid risks. AI writing presents significant opportunities and challenges for document writing. The emergence of new technologies, like twin brothers, presents us with both favorable opportunities and significant challenges.

[0098] List of known errors: redundant editing information, redundant symbols;

[0099] After preliminary cleaning, "(This newspaper, Beijing, February 31) Report / Feedback\n02:37·Xinhua Photo Selection" and "" were deleted.

[0100] S2. Semantic consistency maintenance based on semantic similarity evaluation.

[0101] S3. Middle layer cleaning:

[0102] Field Name: News

[0103] Preliminary Text Cleaning: With the emergence of AI technologies such as ChatGPT, Doubao, Kimi, and DeepSeek, and in today's surging digital age, AI writing, with its rapid text generation, extensive knowledge integration, time-saving costs, and increased writing efficiency, is poised to surpass or even replace traditional document writing. In-depth research has revealed that despite its significant advantages, AI writing cannot, from a theoretical, theoretical, and ethical perspective, truly replace document writing. (Image source: Screenshot of an online AI writing assistant search) AI writing and traditional document writing complement each other and operate in tandem. In the context of AI writing, professionals in organizations must not only discern and adapt to trends, but also proactively capitalize on them, avoid risks, and proactively take action. AI writing presents significant opportunities and challenges for document writing. The emergence of new technologies, like twin brothers, presents us with both promising opportunities and significant challenges.

[0104] List of known errors: redundant editing information;

[0105] The “Image source: Internet AI Writing Assistant search screenshot” was deleted through efficient cleaning.

[0106] S4. Input the text data to be cleaned and the text data after efficient cleaning into the improvement layer model to improve and optimize the text quality, completing the text data cleaning

[0107] The embodiments described above are merely descriptions of preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Without departing from the spirit of the present invention, various modifications and improvements made to the technical solutions of the present invention by persons skilled in the art should fall within the scope of protection defined by the claims of the present invention.

Claims

1. A text data cleaning method based on multi-level large model serialization, characterized in that: The following steps are involved: Acquire text data to be cleaned, and perform error review and preliminary cleaning on the text data to be cleaned using the initial layer large model to obtain preliminary cleaned text data; Maintaining semantic consistency of the text data after preliminary cleaning based on a semantic similarity evaluation mechanism; The middle-layer large model is used to understand unknown errors in the maintained text and efficiently clean it, thus obtaining efficiently cleaned text data. The text data to be cleaned and the text data after efficient cleaning are input into the improvement layer model to improve and optimize the text quality, thereby completing the text data cleaning.

2. The text data cleaning method based on multi-level large model serialization according to claim 1 is characterized in that: The method for obtaining the preliminarily cleaned text data includes: Obtaining a first system prompt word based on the text data to be cleaned and the domain information, and obtaining a first user prompt word through a known error list provided by a user; The initial layer large model performs error review on the text to be cleaned using the first system prompt word and the first user prompt word; The initial layer large model performs preliminary cleaning on the text data after error review to obtain the preliminary cleaned text data.

3. The text data cleaning method based on multi-level large model serialization according to claim 1 is characterized in that: Methods for maintaining semantic consistency include: Mapping the text data to be cleaned and the text data after preliminary cleaning to a high-dimensional semantic space respectively using semantic embedding technology to obtain a vector representation of the text to be cleaned and a vector representation of the text after preliminary cleaning; The similarity between the text vector representation to be cleaned and the text vector after preliminary cleaning is calculated by cosine similarity: Where Score represents the similarity, A represents the text vector after preliminary cleaning, B represents the text vector to be cleaned, · represents the vector dot product, and || || represents the modulus of the vector; Setting a semantic similarity threshold, and comparing the obtained similarity with the semantic similarity threshold; If the obtained similarity is greater than the semantic similarity threshold, there is no need to perform semantic consistency maintenance; If the obtained similarity is less than the semantic similarity threshold, the inference parameters of the initial layer large model are updated and re-cleaned until the iteration reaches a similarity greater than the semantic similarity threshold, completing the semantic consistency maintenance.

4. The text data cleaning method based on multi-level large model serialization according to claim 1 is characterized in that: The method for obtaining the efficiently cleaned text data includes: Use COT thinking chain prompts and structured templates to build the second system prompt words and the second user prompt words; The second system prompt word and the second user prompt word are combined to perform unknown error understanding and efficient cleaning on the maintained text to obtain the efficiently cleaned text data.

5. The text data cleaning method based on multi-level large model serialization according to claim 1 is characterized in that: Methods for improving and optimizing text quality include: Analyzing the difference between the text data to be cleaned and the text data after efficient cleaning by using the improvement layer large model; The text errors in the efficiently cleaned text data are identified and corrected by analyzing the differences, thereby completing the text data cleaning.

6. A text data cleaning system based on multi-level large model serialization, the system applying the method according to any one of claims 1 to 5, characterized in that: Includes: preliminary cleaning module, semantic maintenance module, efficient cleaning module and complete optimization module; The preliminary cleaning module is used to obtain text data to be cleaned, and perform error review and preliminary cleaning on the text data to be cleaned through the initial layer large model to obtain preliminary cleaned text data; The semantic maintenance module performs semantic consistency maintenance on the text data after preliminary cleaning based on a semantic similarity evaluation mechanism; The efficient cleaning module uses the middle layer large model to understand unknown errors and efficiently clean the maintained text to obtain efficiently cleaned text data; The improvement and optimization module is used to input the text data to be cleaned and the text data after efficient cleaning into the improvement layer model to improve and optimize the text quality, thereby completing text data cleaning.

7. A text data cleaning system based on multi-level large model serialization according to claim 6, characterized in that: The workflow of the preliminary cleaning module includes: Obtaining a first system prompt word based on the text data to be cleaned and the domain information, and obtaining a first user prompt word through a known error list provided by a user; The initial layer large model performs error review on the text to be cleaned using the first system prompt word and the first user prompt word; The initial layer large model performs preliminary cleaning on the text data after error review to obtain the preliminary cleaned text data.

8. A text data cleaning system based on multi-level large model serialization according to claim 6, characterized in that: The workflow of the semantic maintenance module includes: Mapping the text data to be cleaned and the text data after preliminary cleaning to a high-dimensional semantic space respectively using semantic embedding technology to obtain a vector representation of the text to be cleaned and a vector representation of the text after preliminary cleaning; The similarity between the text vector representation to be cleaned and the text vector after preliminary cleaning is calculated by cosine similarity: Where Score represents the similarity, A represents the text vector after preliminary cleaning, B represents the text vector to be cleaned, · represents the vector dot product, and || || represents the modulus of the vector; Setting a semantic similarity threshold, and comparing the obtained similarity with the semantic similarity threshold; If the obtained similarity is greater than the semantic similarity threshold, there is no need to perform semantic consistency maintenance; If the obtained similarity is less than the semantic similarity threshold, the inference parameters of the initial layer large model are updated and re-cleaned until the iteration reaches a similarity greater than the semantic similarity threshold, completing the semantic consistency maintenance.

9. A text data cleaning system based on multi-level large model serialization according to claim 6, characterized in that: The workflow of the efficient cleaning module includes: Use COT thinking chain prompts and structured templates to build the second system prompt words and the second user prompt words; The second system prompt word and the second user prompt word are combined to perform unknown error understanding and efficient cleaning on the maintained text to obtain the efficiently cleaned text data.

10. A text data cleaning system based on multi-level large model serialization according to claim 6, characterized in that: The workflow of the improvement and optimization module includes: Analyzing the difference between the text data to be cleaned and the text data after efficient cleaning by using the improvement layer large model; The text errors in the efficiently cleaned text data are identified and corrected by analyzing the differences, thereby completing the text data cleaning.

Images