Text data processing method and device, computer equipment and readable storage medium

Abstract

The invention relates to a text data processing method and device, computer equipment, a computer readable storage medium and a computer program product. The text data cleaning efficiency can be improved, and the accuracy and safety of the text data are guaranteed. The method comprises the steps of obtaining each piece of to-be-cleaned text data, performing duplicate removal processing and abnormal data processing on each piece of text data to obtain each piece of first text data, and then performing risk word rule filtering processing and large language model security detection processing on each piece of first text data to obtain each piece of second text data, and finally, inputting each second text data into an error correction large model, and performing language expression error correction processing on each second text data by the error correction large model to obtain each target text data.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a text data processing method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Art

[0002] With the rapid development of artificial intelligence technology, various industries can now improve work efficiency through artificial intelligence models. Among them, the performance and actual application effect of artificial intelligence models mainly depend on the data used to train the models. However, in actual applications, the collected text data often contains content that does not meet the requirements. For example, it may contain some content that is not suitable for publication or dissemination. The models trained with such text data often have poor question-answering accuracy and weak generalization ability, and may also have security issues.

[0003] In order to ensure the performance and practical application effect of artificial intelligence models, the collected text data needs to be cleaned. However, the amount of model training data is generally at the TB (Terabyte) or PB (Petabyte) level, and manual data cleaning is difficult. There are technical problems such as low efficiency and accuracy of text data cleaning. Summary of the invention

[0004] Based on this, it is necessary to provide a text data processing method, device, computer equipment, computer-readable storage medium and computer program product to address the above technical problems.

[0005] In a first aspect, the present application provides a text data processing method, the method comprising:

[0006] Get each text data to be cleaned;

[0007] Performing deduplication processing and abnormal data processing on each of the text data to obtain each first text data;

[0008] Perform risk word rule filtering processing and large language model security detection processing on each of the first text data to obtain each second text data;

[0009] The respective second text data are input into a large error correction model, and the large error correction model performs language expression error correction processing on the respective second text data to obtain respective target text data.

[0010] In a second aspect, the present application provides a text data processing device, the device comprising:

[0011] A text data acquisition module, used to acquire each text data to be cleaned;

[0012] A deduplication and exception processing module, used for performing deduplication processing and exception data processing on each of the text data to obtain each first text data;

[0013] A filtering and detection module, used for performing risk word rule filtering processing and large language model security detection processing on each of the first text data to obtain each second text data;

[0014] The error correction processing module is used to input the respective second text data into the error correction large model, and the error correction large model performs language expression error correction processing on the respective second text data to obtain respective target text data.

[0015] In a third aspect, the present application provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above method when executing the computer program.

[0016] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the steps of the above method when executed by a processor.

[0017] In a fifth aspect, the present application provides a computer program product, including a computer program, which implements the steps of the above method when executed by a processor.

[0018] The above-mentioned text data processing method, device, computer equipment, computer-readable storage medium and computer program product obtain each text data to be cleaned, perform deduplication and abnormal data processing on each text data to obtain each first text data, then perform risk word rule filtering processing and large language model security detection processing on each first text data to obtain each second text data, and finally input each second text data into the error correction large model, and the error correction large model performs language expression error correction processing on each second text data to obtain each target text data. This scheme first performs deduplication and abnormal data processing on each text data to be cleaned, and then performs risk word rule filtering processing and large language model security detection processing to ensure that the text data has no security issues. Finally, the text data without security issues is subjected to language expression error correction processing by the error correction large model to obtain target text data that can be used for model training, thereby improving the efficiency of text data cleaning processing and ensuring the accuracy and security of text data. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0020] Figure 1 A flowchart of a text data processing method in one embodiment;

[0021] Figure 2 is a flowchart of a text data processing method in another embodiment;

[0022] Figure 3 is a structural block diagram of a text data processing device in one embodiment;

[0023] Figure 4 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0024] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0025] Introduction to relevant terms involved in this application:

[0026] Rule strategy: a method of processing text data through some rule scripts or simple algorithms.

[0027] Big security model: various risk identification models specifically designed to identify each type of risk, such as a risk identification model used to identify various types of content that should not be published or disseminated.

[0028] For the large question-answer detection model, input the input data or question data into the model for the trained question-answer pair data or fine-tuning data to check whether the actual output of the large question-answer detection model contains any non-safety text data.

[0029] Big model error correction: The process of using a general big model or a professional error correction big model to correct the syntax, semantics, and expression of text data.

[0030] In one embodiment, Figure 1 As shown, a text data processing method is provided, which can be executed by a server or a terminal, and the method can include the following steps:

[0031] Step S101, obtaining each text data to be cleaned.

[0032] Among them, communication business types can currently process business texts through artificial intelligence models. The artificial intelligence model that can process business texts needs to be trained with text data that meets performance requirements. In order to make the text data involved in training more effective, the text data needs to be cleaned.

[0033] Among them, before using text data to train the model, the text data needs to be cleaned before training to ensure that the text data is accurate, consistent, non-duplicative, safe and reliable, thereby improving the actual use effect of artificial intelligence model training and the accuracy and security of the model.

[0034] In this step, various text data to be cleaned can be obtained. Among them, the text data can be data in the form of documents, etc., and the text data can contain text content, and the text content can be, for example, various characters in the document. The number of text data obtained can be multiple. Among them, each text data can contain text content, and each text content can include multiple characters. In this regard, in subsequent steps, data cleaning can be performed according to relevant rule strategies, such as deduplication of text, abnormal data processing of text (such as correction, removal of abnormal characters or abnormal encodings, etc.), and risk word rules and large language models can be applied to perform content security review processing, and error correction processing of language expression of text can be performed through error correction large models, etc.

[0035] Step S102, performing deduplication processing and abnormal data processing on each text data to obtain each first text data.

[0036] Among them, since the data volume of the above text data is very large, for the large-volume batch data cleaning work, the text data can be bucketed and processed by concurrent streaming. For example, data bucketing and concurrent streaming processing can be performed through tools such as flink, Alink, sparkstream, etc., thereby processing batches of text data.

[0037] In this step, after obtaining each text data, deduplication processing and abnormal data processing can be performed on each text data, so as to obtain each first text data. The deduplication processing can include multi-level deduplication processing, for example, deduplication between different documents and deduplication within the same document. After deduplication processing is performed on each text data, abnormal data processing, such as correction and removal, can be performed. After the deduplication processing and abnormal data processing are performed on each text data, each first text data is obtained.

[0038] For step S102, in one embodiment, step S102 may include: for every two text data, if the text content similarity of the two text data is greater than a preset similarity threshold, then determine the retained text data from the two text data according to the text content corresponding to the two text data, and for each retained text data, deduplicate the characters in the retained text data to obtain each deduplicated text data; for each deduplicated text data, obtain Chinese data in the text data; remove abnormal codes and abnormal characters in the Chinese data, and convert the abnormal text format in the Chinese data into a preset text format; obtain each first text data based on the Chinese data in the preset text format without the abnormal codes and abnormal characters.

[0039] This embodiment includes a method for deduplication processing and abnormal data processing for each text data. First, deduplication processing is performed on each text data, and then abnormal data processing is performed on each deduplication processed text data, so as to obtain each first text data.

[0040] Wherein, for deduplication processing of each text data, deduplication processing may include multiple levels of deduplication processing, for example, deduplication between different documents and deduplication within the same document, etc. There are multiple text data, each of which may be a document, and the terminal may perform deduplication processing on the text data in units of two text data. Wherein, for each two text data, the terminal may detect the similarity of the text content of the two text data. Wherein, if it is detected that the text content similarity of the two text data is less than or equal to a preset similarity threshold, it may be determined that the two text data are not the same text, and the two text data may be retained, that is, both may be used as retained text data. If it is detected that the text content similarity of the two text data is greater than the preset similarity threshold, it may be determined that the two text data are similar text data, and one of the two text data may be selected as the retained text data according to the text content corresponding to the two text data. As an available example, the text data with better text quality may be selected from the two text data for retention, etc. Thus, multiple retained text data may be obtained. For each retained text data, the content in the retained text data can be deduplicated. As an example, the terminal can perform similarity detection on each character in the retained text data, and deduplicate characters whose similarity is greater than a preset similarity threshold. Thus, each deduplicated text data can be obtained.

[0041] Regarding the determination of the retained text data, further, in one embodiment, the above-mentioned determination of the retained text data from the two text data according to the text contents corresponding to the two text data may include:

[0042] The two text data are input into a scoring model, and the scoring model outputs scores corresponding to the two text data based on the text contents of the two text data; and the retained text data is determined from the two text data according to the scores.

[0043] In this embodiment, for two text data determined to be similar in content, the terminal can retain one of the text data as the retained text data. Among them, the scoring model to be trained can be iteratively trained in advance using multiple text data samples to obtain a trained scoring model. The scoring model can be a neural network model. The terminal can input the above two text data determined to be similar in content into the trained scoring model. The scoring model can be used to perform content scoring based on the text content corresponding to each of the two text data, for example, scoring based on the text standard in the above communication field, so as to obtain and output the scores corresponding to each of the two text data. In this way, the retained text data can be determined from the two texts based on the scores corresponding to each of the above two text data. As an example, the text data with a higher score in the two text data can be used as the retained text data. Among them, if the scores are the same, the text data with a later storage time in the two text data can be used as the retained text data.

[0044] In practical applications, the above-mentioned various text data can be deduplicated in batches, including removing similar document content and deduplicating similar content in documents. Among them, each text data can be a document. If it is detected that the document names are the same or different, but the part of the same text content in the document is greater than a preset similarity threshold, for example, greater than 90%, then the two documents can be considered duplicates, and one document can be selected as the retained text data. After deduplicating the document, similar content in the document can be deduplicated, for example, based on the recognition of similar characters in the document, deduplication within the document can be achieved, thereby achieving compression of the document content and ensuring that the training data is free of duplicate data.

[0045] Among them, the algorithms used for deduplication between documents include but are not limited to Minhash+LSH (Min Hashing+Locality Sensitive Hashing), SimHash (Similarity-Hash), minHash (Min Hashing), etc. The algorithms used for deduplication within text include but are not limited to various similarity algorithms such as tfidf (term frequency–inverse document frequency), word2vec9 (word to vector), text2vec (text to vector), etc.

[0046] Through the above embodiments, the text data can be deduplicated including deduplication between documents and deduplication of the content of the document itself, and the better documents can be selected for retention, thereby improving the accuracy and efficiency of the text data cleaning process.

[0047] Among them, for the abnormal data processing of each deduplicated text data, the text data may have different sources, and the text data from different sources may include a variety of text contents and formats, etc., and there may also be language mixing, encoding anomalies, etc., so the abnormal data processing can be performed on the first text data after deduplication.

[0048] In this embodiment, for each deduplication-processed text data, the terminal can obtain Chinese data in the text data. Among them, the Chinese data in the text data can be extracted by extracting the text in a specific language. In some embodiments, the abnormal encoding, abnormal characters and abnormal text formats in the Chinese data can be determined according to the preset encoding rules. Among them, the preset encoding rules can be the encoding rules of the platform where the artificial intelligence model trained using the text data is located; the abnormal encoding can be the encoding that does not meet the preset encoding rules; the abnormal characters can be the characters that do not meet the preset encoding rules; the abnormal text format can be the text format that does not meet the preset encoding rules.

[0049] Then, the abnormal codes and abnormal characters in the Chinese data can be removed, and the abnormal text format in the Chinese data can be converted into a preset text format. Among them, the preset text format can be the text format corresponding to the platform where the above-mentioned artificial intelligence is located. Then, each first text data can be obtained based on the Chinese data in the preset text format with the abnormal codes and abnormal characters removed. That is, each first text data can be Chinese data in a preset text format, and does not contain abnormal codes and abnormal characters that do not meet the preset coding rules of the platform where the above-mentioned artificial intelligence is located. Based on this, each first text data can be processed through subsequent steps such as content security review and language expression error correction.

[0050] In actual applications, the text data used for model training comes from various websites, and their document formats and contents are various. There may be abnormal encodings, mixing of various languages, etc. Therefore, the abnormal encoding characters can be corrected or removed, and the text format can be unified. The text language can be extracted on demand. After extracting the text data in the required language, for example, after extracting Chinese data, the abnormal encoding characters of the text data can be corrected, the text format can be unified, and other processing can be achieved based on the above preset encoding rules.

[0051] Step S103 , performing risk word rule filtering processing and large language model security detection processing on each first text data to obtain each second text data.

[0052] Among them, the risk word rule filtering process can be filtering the text data by using the preset rules containing risk words, such as removing the text data containing risk words, etc. The large language model security detection process can be performing security detection on the text data by using the large language model, such as identifying the text data containing the content that is not suitable for publication or dissemination, etc.

[0053] Among them, custom risk words can be pre-built for risk word rule filtering processing. Among them, risk words can include custom risk keywords and custom blacklists, etc. Various types of risk identification models and question-and-answer detection models can also be deployed for large language model security detection processing. Among them, various types of risk identification models can include a large language model for performing security detection processing on text data on various types of content that are not suitable for publication and dissemination, and the question-and-answer detection model can include a large language model for performing security detection processing on text data in the format of question-and-answer pairs. In actual applications, risk identification models and question-and-answer detection models can be derived from secondary fine-tuning training of open source models based on risk vocabulary.

[0054] In this step, each first text data may be subjected to risk word rule filtering processing and large language model security detection processing in turn to obtain each second text data, thereby realizing content security review of the text data.

[0055] Step S104, input each second text data into the error correction large model, and the error correction large model performs language expression error correction processing on each second text data to obtain each target text data.

[0056] The error correction model to be trained can be iteratively trained based on specific text data samples in advance to obtain a trained error correction model. The error correction model can be a model that performs language expression error correction processing on text data. For example, the text data obtained above may contain text data that does not meet the semantics and grammatical structure of the communication field. The error correction model can be used to identify the text data that does not meet the semantics and grammatical structure of the communication service type and perform correction and other processing.

[0057] In this step, each second text data can be input into the trained error correction model, and the error correction model can perform language expression error correction processing on each second text data, and can identify abnormal language expressions in the second text data, for example, it can identify the second text data that does not meet the semantics and grammatical structure of the communication service type, and then the semantics and grammatical structure of these second text data can be corrected to obtain each target text data. Among them, each target text data can be text data after data cleaning, so that the artificial intelligence model to be trained can be trained based on these target text data, so that the trained artificial intelligence model can be used to perform business processing of related communication service types.

[0058] The text data processing method of the above embodiment obtains each text data to be cleaned, performs deduplication and abnormal data processing on each text data to obtain each first text data, then performs risk word rule filtering processing and large language model security detection processing on each first text data to obtain each second text data, and finally inputs each second text data into the error correction large model, and the error correction large model performs language expression error correction processing on each second text data to obtain each target text data. This solution first performs deduplication and abnormal data processing on each text data to be cleaned, then performs risk word rule filtering processing and large language model security detection processing to ensure that the text data has no security issues, and finally performs language expression error correction processing on the text data without security issues through the error correction large model to obtain target text data that can be used for model training, thereby improving the efficiency of text data cleaning processing and ensuring the accuracy and security of text data.

[0059] In some embodiments, performing risk word rule filtering and large language model security detection processing on each first text data in step S103 to obtain each second text data may include:

[0060] According to the preset risk word rules, the risk words in each first text data are removed to obtain each first text data after being filtered and processed by the risk word rules; each first text data after being filtered and processed by the risk word rules is input into the multi-category risk identification model in turn, and the multi-category risk identification model is used to remove various types of risk text data in each first text data after being filtered and processed by the risk word rules to obtain each first text data after being filtered and processed by the risk model; it is determined whether the data format of the first text data after being filtered and processed by the risk model is a question-answer pair format, and if so, for each group of question-answer pairs in the first text data, the abnormal question-answer pairs in each group of question-answer pairs are removed by the question-answer detection model to obtain each second text data.

[0061] In this embodiment, risk words are first removed from each first text data according to preset risk word rules, and then each first text data filtered and processed by the risk word rules is sequentially input into a multi-category risk identification model to filter each type of risk text data therein, and finally, abnormal question and answer pairs in the first text data in the question and answer pair format are removed through the question and answer detection model to obtain each second text data.

[0062] Among them, for removing risk words from each first text data according to the preset risk word rules, the preset risk word rules can be constructed based on the text content in the custom risk keyword table and the blacklist, and can be used to identify text data that expresses risks. Therefore, based on the custom abnormal keywords and blacklist data in the preset risk word rules, the risk words in each first text data can be identified, and the risk words in each first text data can be removed to obtain each first text data after filtering and processing by the risk word rules. Among them, the preset risk word rules can be updated in real time to add new risk words, such as adding some network terms or industry slang, etc., to ensure the flexibility of the system.

[0063] Among them, for each first text data after the risk word rule filtering process, the various risk text data are sequentially input into the multi-category risk identification model to filter the various risk text data therein, and different categories of risk identification models can be used to identify the risks of different categories of text content that is not suitable for release, dissemination, etc., and the first text data containing the text content that is not suitable for release, dissemination, etc. can be recorded as risk text data, and different categories of risk text data are various types of risk text data. In this embodiment, each first text data after the risk word rule filtering process is sequentially input into the multi-category risk identification model, and each type of risk identification model can remove the corresponding category of risk text data in each first text data after the risk word rule filtering process, so that the various types of risk text data in each first text data after the risk word rule filtering process can be removed by the multi-category risk identification model, and each first text data after the risk model filtering process is obtained.

[0064] Among them, for obtaining each second text data by removing abnormal question and answer pairs in the first text data whose data format is the question and answer pair format through the question and answer detection model, after obtaining each first text data after the risk model filtering process, for each first text data, it can be determined whether the data format of the first text data is the question and answer pair format. For the first text data whose data format is the question and answer pair format, each group of question and answer pairs in the first text data can be obtained, and each group of question and answer pairs can include question data and corresponding answer data. Then, the question and answer detection model can be used to detect whether each group of question and answer pairs is an abnormal question and answer pair, wherein the abnormal question and answer pair can be the question data or answer data in the question and answer pair that have an abnormality, and the abnormality can be the text content that is not suitable for publication, dissemination, etc. Therefore, the abnormal question and answer pairs in each group of question and answer pairs can be removed through the question and answer detection model to obtain each second text data.

[0065] Through the solution of this embodiment, the accuracy and security of the text data finally used for model training can be ensured through risk word rules, multi-category risk identification models and question-answer detection models in sequence.

[0066] In one embodiment, in the above embodiment, removing abnormal question-answer pairs in each group of question-answer pairs by using the question-answer detection model to obtain each second text data may include:

[0067] The question data in the question and answer pair is input into the question and answer detection model, and the question and answer detection model outputs the model answer data according to the question data. If there is a risk in the model answer data, the question and answer pair is removed as an abnormal question and answer pair; the question data and answer data in the question and answer pair are respectively input into the question and answer detection model, and the question and answer detection model performs risk detection on the question data and the answer data respectively. If there is a risk in the question data or the answer data, the question and answer pair is removed as an abnormal question and answer pair; and the second text data is obtained based on the first text data after removing the abnormal question and answer pairs.

[0068] This embodiment mainly uses the question-answer detection model to identify and remove abnormal question-answer pairs. The identification of abnormal question-answer pairs includes two identification processes. First, the question data in the question-answer pair can be input into the question-answer detection model, and the question-answer detection model outputs the model answer data according to the question data, that is, the question-answer detection model outputs an answer data (model answer data) according to the input question data, and then the question-answer detection model can detect the model answer data. If it is detected that the model answer data has risks, such as containing relevant content that is not suitable for release or dissemination, the question-answer pair can be removed as an abnormal question-answer pair. Secondly, the question data and answer data in the question-answer pair can be input into the question-answer detection model respectively, and the question-answer detection model can perform risk detection on the question data and the answer data respectively. The question-answer detection model can detect whether the question data and the answer data have text security risks, such as whether they contain relevant content that is not suitable for release or dissemination. In this regard, whether the question data or the answer data has risks, the question-answer detection model will remove the question-answer pair as an abnormal question-answer pair.

[0069] Through the above-mentioned identification and removal of abnormal question and answer pairs, each first text data after removing the abnormal question and answer pairs can be obtained. The each first text data can be used as each second text data for subsequent model training, so as to further improve the security and accuracy of the model training data.

[0070] In some embodiments, the step S104 of inputting each second text data into the error correction big model, and performing language expression error correction processing on each second text data by the error correction big model to obtain each target text data may include:

[0071] Each second text data is input into the error correction big model, and the error correction big model identifies whether there is a language expression error in each second text data; the language expression error may include one or more of grammatical errors, semantic ambiguity and expression abnormality; if so, the error correction big model corrects the second text data with the corresponding language expression error, and outputs each corrected second text data, and stores each corrected second text data and the corresponding correction record in the cloud or locally; based on each corrected second text data and the corresponding correction record in the cloud or locally, each target text data is obtained.

[0072] This embodiment also uses a large error correction model to correct errors and form the final target text data that can be used for model training, thereby improving the standardization and accuracy of the model training data.

[0073] In this embodiment, each second text data can be input into the error correction big model, and the error correction big model is used to perform language expression error correction processing on each second text data. The language expression error correction processing can be to identify whether there is a language expression error and correct it. The language expression error can include one or more of grammatical errors, semantic ambiguity and expression abnormality, such as grammatical disorder, inaccurate semantic expression, ambiguity, spelling errors, conventional expression abnormality, professional terminology errors, place name identification errors, misidentification of names, etc. In this regard, the error correction model can identify whether there are language expression errors in each second text data. If so, the error correction model can correct the second text data with the corresponding language expression errors and output the corrected second text data. It can also output the corresponding correction records (such as detailed error logs and correction records of corrections, etc.), and then the corrected second text data and the corresponding correction records can be stored in the cloud or locally. Therefore, based on the corrected second text data and the corresponding correction records in the cloud or locally, each target text data can be finally obtained. For example, the corrected second text data and the correction records in the cloud or locally can be combined to correct the corrected second text data again (which can be manual correction), so as to obtain the final target text data that can be used for model training.

[0074] The solution of this embodiment can identify and correct some language expression errors that may still exist in the text data after deduplication and abnormal data processing, risk word rule filtering processing and large language model security detection processing, with the help of a large error correction model, thereby improving the standardization, accuracy and efficiency of the text data used for model training.

[0075] As an application example, Figure 2 As shown, a text data processing method is also provided. In this application example, a large amount of text data can be processed in batches and buckets in parallel. Each batch of data is first deduplicated, and then the data in each text is corrected to remove special characters. The data cleaned by the above rule strategy is audited for security to ensure that there are no security issues in the training data. Finally, the audited data is corrected for errors in the large model and finally stored.

[0076] First, the text data can be deduplicated. The terminal can perform text deduplication in batches, mainly to remove similar document content. For example, if the document names are different but more than 90% of the content in the document is the same, the document can be considered duplicate, and the system selects a document to retain as the retained text data. After deduplication between documents, the terminal deduplicates similar content in the document and obtains deduplicated text data, thereby achieving document content compression and ensuring that the training data does not contain duplicate data. The algorithms for deduplication between documents usually include Minhash+LSH, SimHash, minHash, etc., and deduplication within text usually uses various similarity algorithms such as tfidf, word2vec, text2vec, etc.

[0077] Then, the abnormal data in the deduplicated text data can be corrected or removed. Wherein, the training text data comes from various websites, the document content formats are various, or the document content has abnormal coding and various languages ​​mixed, then the terminal can correct or remove the abnormal coding characters in the deduplicated text data, unify the text format, extract the text language as needed, such as extracting Chinese data, etc., thereby obtaining the first text data after deduplication and abnormal data processing.

[0078] Then, the first text data can be subjected to risk word rule filtering processing and large language model security detection processing. Among them, the risk words in each first text data can be removed according to the preset risk word rules to obtain each first text data after risk word rule filtering processing. The risk word rules can include risk keyword filtering and blacklist text data judgment, and the text data that hits the risk keyword or hits the blacklist can be discarded. Then, each first text data after risk word rule filtering processing can be sequentially input into a multi-class risk identification model, and the multi-class risk identification model removes various types of risk text data in each first text data after risk word rule filtering processing to obtain each first text data after risk model filtering processing, that is, the relevant text data detected by the risk identification model can continue to be detected by the question and answer detection model, and the relevant text data that fails to pass the risk identification model detection is discarded. Then, it can be determined whether the data format of the first text data after risk model filtering is in the question-answer pair format. If so, each group of question-answer pairs in the first text data is removed from the abnormal question-answer pairs through the question-answer detection model to obtain each second text data. That is, the relevant text data detected by the question-answer detection model can pass the content security review, and the relevant text data that fails to pass the question-answer detection model will be discarded.

[0079] After deduplication and abnormal data processing, risk word rule filtering processing and large language model security detection processing, some language expression errors may still exist in the text data, which can finally be identified and corrected with the help of the error correction model, which may include but is not limited to grammatical correction, semantic correction, professional terminology correction and general expression correction. Among them, the error correction model can be a general large model or a professional error correction model related to the business. The error correction model can divide the language structure processing algorithm based on the business field to which the input text data belongs, and select the corresponding professional error correction algorithm for language structure processing.

[0080] This application example can combine rule strategies with various large security models and error correction models to perform efficient and accurate data cleaning on model training data, which can ensure the accuracy, consistency, high availability and security of training data, thereby improving the actual application effect of the final trained model and the generalization ability of the model while ensuring the security of model use.

[0081] It should be understood that, although the steps in the flowcharts involved in the above embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0082] Based on the same inventive concept, the embodiment of the present application also provides a text data processing device for implementing the text data processing method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more text data processing device embodiments provided below can refer to the limitations of the text data processing method above, and will not be repeated here.

[0083] In an exemplary embodiment, Figure 3 As shown, a text data processing device is provided, and the device 300 may include:

[0084] The text data acquisition module 301 is used to acquire each text data to be cleaned;

[0085] A deduplication and exception processing module 302 is used to perform deduplication processing and exception data processing on each text data to obtain each first text data;

[0086] A filtering and detection module 303 is used to perform risk word rule filtering processing and large language model security detection processing on each of the first text data to obtain each second text data;

[0087] The error correction processing module 304 is used to input the respective second text data into the error correction large model, and the error correction large model performs language expression error correction processing on the respective second text data to obtain respective target text data.

[0088] In one embodiment, the deduplication and exception handling module 302 is used to, for each pair of the text data, if the similarity of the text contents of the two text data is greater than a preset similarity threshold, determine the retained text data from the two text data according to the text contents corresponding to the two text data, and for each retained text data, deduplicate the characters in the retained text data to obtain each deduplicated text data; for each deduplicated text data, obtain Chinese data in the text data; remove abnormal codes and abnormal characters in the Chinese data, and convert the abnormal text format in the Chinese data into a preset text format; and obtain each first text data based on the Chinese data in the preset text format without the abnormal codes and abnormal characters.

[0089] In one embodiment, the deduplication and exception handling module 302 is used to input the two text data into a scoring model, and the scoring model outputs the scores corresponding to the two text data based on the text contents of the two text data; and determines the retained text data from the two text data according to each of the scores.

[0090] In one embodiment, the deduplication and exception handling module 302 is further used to determine abnormal codes, abnormal characters and abnormal text formats in the Chinese data according to preset encoding rules.

[0091] In one embodiment, the filtering and detection module 303 is used to remove risk words from each of the first text data according to a preset risk word rule, so as to obtain each of the first text data after being filtered and processed by the risk word rule; input each of the first text data after being filtered and processed by the risk word rule into a multi-category risk identification model in turn, and use the multi-category risk identification model to remove various types of risk text data from each of the first text data after being filtered and processed by the risk word rule, so as to obtain each of the first text data after being filtered and processed by the risk model; determine whether the data format of the first text data after being filtered and processed by the risk model is a question-answer pair format, and if so, remove abnormal question-answer pairs from each group of question-answer pairs in the first text data through a question-answer detection model to obtain each of the second text data.

[0092] In one embodiment, the filtering and detection module 303 is used to input the question data in the question and answer pair into the question and answer detection model, and the question and answer detection model outputs model answer data based on the question data. If the model answer data is at risk, the question and answer pair is removed as an abnormal question and answer pair; the question data and answer data in the question and answer pair are respectively input into the question and answer detection model, and the question and answer detection model performs risk detection on the question data and answer data respectively. If the question data or the answer data is at risk, the question and answer pair is removed as an abnormal question and answer pair; and the second text data is obtained based on the first text data after removing the abnormal question and answer pairs.

[0093] In one embodiment, the error correction processing module 304 is used to input the respective second text data into the error correction big model, and the error correction big model identifies whether there are language expression errors in the respective second text data; the language expression errors include one or more of grammatical errors, semantic ambiguity and expression anomalies; if so, the error correction big model corrects the second text data with the corresponding language expression errors, and outputs the corrected respective second text data, and stores the corrected respective second text data and the corresponding correction records in the cloud or locally; and obtains the respective target text data according to the corrected respective second text data and the corresponding correction records in the cloud or locally.

[0094] Each module in the above-mentioned text data processing device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each module.

[0095] In an exemplary embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as shown in FIG. Figure 4As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device can be used to store text data, etc. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a text data processing method is implemented.

[0096] Those skilled in the art will understand that Figure 4 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0097] In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the above-mentioned text data processing method when executing the computer program.

[0098] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the above-mentioned text data processing method is implemented.

[0099] In one embodiment, a computer program product is provided, including a computer program, which implements the above-mentioned text data processing method when executed by a processor.

[0100] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0101] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0102] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0103] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A text data processing method, characterized in that: The method comprises: Get each text data to be cleaned; Performing deduplication processing and abnormal data processing on each of the text data to obtain each first text data; Perform risk word rule filtering processing and large language model security detection processing on each of the first text data to obtain each second text data; The respective second text data are input into a large error correction model, and the large error correction model performs language expression error correction processing on the respective second text data to obtain respective target text data.

2. The method according to claim 1, characterized in that The performing deduplication processing and abnormal data processing on each of the text data to obtain each of the first text data includes: For each pair of the text data, if the text content similarity between the two text data is greater than a preset similarity threshold, determining the retained text data from the two text data according to the text content corresponding to the two text data, and for each retained text data, performing deduplication processing on each character in the retained text data to obtain each text data after deduplication processing; For each deduplicated text data, Chinese data in the text data is obtained; abnormal codes and abnormal characters in the Chinese data are removed, and the abnormal text format in the Chinese data is converted into a preset text format; and each first text data is obtained based on the Chinese data in the preset text format without abnormal codes and abnormal characters.

3. The method according to claim 2, characterized in that The step of determining the retained text data from the two text data according to the text contents corresponding to the two text data comprises: Inputting the two text data into a scoring model, and having the scoring model output scores corresponding to the two text data based on the text contents of the two text data; According to the respective scores, the remaining text data is determined from the two text data.

4. The method according to claim 2, characterized in that: Before removing abnormal codes and abnormal characters in the Chinese data and converting the abnormal text format in the Chinese data into a preset text format, the method further includes: According to preset encoding rules, abnormal encodings, abnormal characters and abnormal text formats in the Chinese data are determined.

5. The method according to claim 1, characterized in that The performing risk word rule filtering processing and large language model security detection processing on each of the first text data to obtain each of the second text data includes: According to a preset risk word rule, the risk words in each of the first text data are removed to obtain each of the first text data after being filtered by the risk word rule; Inputting each first text data after risk word rule filtering processing into a multi-class risk identification model in sequence, removing each type of risk text data from each first text data after risk word rule filtering processing by the multi-class risk identification model, and obtaining each first text data after risk model filtering processing; Determine whether the data format of the first text data after risk model filtering is in the question-answer pair format. If so, remove abnormal question-answer pairs in each group of question-answer pairs in the first text data through the question-answer detection model to obtain each second text data.

6. The method according to claim 5, characterized in that The step of removing abnormal question-answer pairs in each group of question-answer pairs by using the question-answer detection model to obtain each second text data includes: Input the question data in the question-answer pair into the question-answer detection model, and the question-answer detection model outputs model answer data according to the question data. If there is a risk in the model answer data, the question-answer pair is removed as an abnormal question-answer pair; The question data and the answer data in the question-answer pair are respectively input into the question-answer detection model, and the question-answer detection model performs risk detection on the question data and the answer data respectively. If there is a risk in the question data or the answer data, the question-answer pair is removed as an abnormal question-answer pair; The second text data are obtained according to the first text data after removing abnormal question-answer pairs.

7. The method according to any one of claims 1 to 6, characterized in that: The step of inputting each second text data into the error correction big model, and performing language expression error correction processing on each second text data by the error correction big model to obtain each target text data includes: Inputting each of the second text data into the error correction model, and using the error correction model to identify whether there are language expression errors in each of the second text data; the language expression errors include one or more of grammatical errors, semantic ambiguity, and expression abnormality; If so, the error correction model corrects the second text data with the corresponding language expression errors, outputs each corrected second text data, and stores each corrected second text data and the corresponding correction record in the cloud or locally; According to the corrected second text data and the corresponding correction records in the cloud or locally, each target text data is obtained.

8. A text data processing device, characterized in that: The device comprises: A text data acquisition module, used to acquire each text data to be cleaned; A deduplication and exception processing module, used for performing deduplication processing and exception data processing on each of the text data to obtain each first text data; A filtering and detection module, used for performing risk word rule filtering processing and large language model security detection processing on each of the first text data to obtain each second text data; The error correction processing module is used to input the respective second text data into the error correction large model, and the error correction large model performs language expression error correction processing on the respective second text data to obtain respective target text data.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images