Text data set cleaning method, device and medium

Abstract

The present invention discloses a text dataset cleaning method, device, and medium, belonging to the field of computer data technology. The method comprises: receiving a text dataset to be cleaned; performing a Chinese filtering operation on a first-format text dataset and a second-format text dataset to form a first-format first text dataset and a second-format first text dataset, respectively; converting the second-format first text dataset into a first-format second text dataset; performing encoding format conversion on the first-format first text dataset and the first-format second text dataset to form a first-format third text dataset; and performing a data cleaning operation on the first-format third text dataset. The present invention implements an efficient data classification and cleaning solution based on Bash scripts and the Spark open source cluster computing framework, capable of processing TB-level Internet text datasets in a short period of time, thereby improving cleaning efficiency.

Description

Technical Field

[0001] The present invention relates to the field of computer data technology, and in particular to a text data set cleaning method, device and medium. Background Art

[0002] In recent years, the internet has seen an increasing number of text datasets, including news, blogs, forums, and e-books. Leveraging these massive amounts of text to generate high-quality datasets for training and reasoning AI models has become a hot research topic. To cleanse these massive text datasets and generate high-quality data sets, numerous data cleaning frameworks have emerged, such as the MapReduce computing framework of the open-source software framework Hadoop and the Spark framework.

[0003] Hadoop is an open-source software framework that supports data-intensive distributed applications. It runs on large clusters built on commodity hardware. Hadoop is a self-implemented implementation based on Google's papers on the MapReduce computing framework and the Google File System.

[0004] The Spark framework is an open source cluster computing framework that uses in-memory computing technology and can perform analytical operations in memory before the data is written to the hard disk.

[0005] However, due to the huge amount of data, many data cleaning solutions are not efficient. Although these data cleaning solutions use parallel computing frameworks such as MapReduce and Spark, the overall design of the data cleaning system and the cleaning algorithms used in each module are relatively inefficient. Therefore, these cleaning systems cannot complete the TB (10 12 ) level data cleaning. Summary of the Invention

[0006] In order to overcome the above technical defects, the purpose of the present invention is to provide a text data set cleaning method, device and medium, which can complete TB (10 12)-level data cleaning. The data source for this cleaning is various Chinese and English e-books downloaded from electronic network drives or e-book websites. A 4TB dataset of internet e-books was cleaned using six servers with 128 CPU cores, 512GB of memory, and a 1.8TB NVMe SSD hard drive. This ultimately generated a 655GB high-quality Chinese e-book dataset. The above method enables an efficient data classification and cleaning solution based on Bash scripts and the Spark open-source cluster computing framework. This solution can process terabyte-level internet text datasets in a short period of time, improving cleaning efficiency.

[0007] The specific technical solutions provided by the embodiments of the present invention are as follows:

[0008] In a first aspect, a method for cleaning a text dataset is provided, the method comprising:

[0009] Receive the text dataset to be cleaned;

[0010] Classifying the text data set to be cleaned into a first format text data set and a second format text data set based on a Bash script;

[0011] Performing a Chinese filtering operation on the first format text dataset and the second format text dataset to form a first format first text dataset and a second format first text dataset, respectively;

[0012] converting the second-format first text dataset into the first-format second text dataset;

[0013] Converting the first text dataset in the first format and the second text dataset in the first format into encoding formats to form a third text dataset in the first format;

[0014] A data cleaning operation is performed on the first format third text data set, wherein the data cleaning operation includes: calculating the first format third text data set using a message digest algorithm and performing a content deduplication operation.

[0015] Furthermore, performing Chinese filtering operations on the first format text dataset and the second format text dataset to form the first format first text dataset and the second format first text dataset respectively includes:

[0016] determining whether the text data in the first format text data set is Chinese text data based on whether the full path of each text data set includes a first preset vocabulary, and if so, deleting the text data; if not, retaining the text data; and / or determining whether the text data is Chinese text data based on whether the first 100 lines of each text data set in the first format text data set include Chinese characters, and if so, retaining the text data; if not, deleting the text data, to form a first format first text data set;

[0017] Whether the text data is Chinese text data is determined based on whether the full path of each text data in the second format text data set includes the first preset vocabulary. If so, it is deleted; if not, it is retained to form the second format first text data set.

[0018] Furthermore, the converting the encoding format of the first format first text dataset and the first format second text dataset to form the first format third text dataset includes:

[0019] The text data in the first format first text data set and the first format second text data set, whose encoding format is the second encoding format, are converted into text data in the first encoding format to form a first format third text data set.

[0020] Furthermore, before calculating the first format third text data set using a message digest algorithm and performing content deduplication operations, the method further includes:

[0021] The first format third text dataset is divided into a first quality text dataset and a second quality text dataset by using a BERT BASE model;

[0022] Delete the second quality text dataset.

[0023] Furthermore, the performing of data cleaning operation on the first format third text dataset further includes:

[0024] Determine whether the number of Chinese characters in each text data in the first format third text data set is greater than or equal to a first threshold and whether the proportion of Chinese characters in the text data is greater than or equal to a second threshold;

[0025] If yes, keep the text data; if no, delete it.

[0026] Furthermore, the performing of data cleaning operation on the first format third text dataset further includes:

[0027] The collected sensitive words are divided into the first category of sensitive words and the second category of sensitive words;

[0028] Traversing the first format third text data set;

[0029] If the text data in the first format third text data set includes the first category of sensitive words, deleting them;

[0030] If the text data in the first format third text data set includes the second category of sensitive words, the paragraph including the second category of sensitive words is deleted, and the rest of the text data is retained.

[0031] Furthermore, the performing of data cleaning operation on the first format third text dataset further includes:

[0032] removing special characters from each text data in the first format third text data set;

[0033] The traditional Chinese characters of each text data in the first format third text data set are converted into simplified Chinese characters.

[0034] Furthermore, the text dataset cleaning method includes:

[0035] Performing sampling verification on the cleaned first format third text data set;

[0036] The first format third text data set that passes the inspection is stored.

[0037] In a second aspect, a text data set cleaning device is provided, the device comprising:

[0038] A memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the text dataset cleaning method as described in any one of the first aspects.

[0039] In a third aspect, a computer storage medium is provided, the medium comprising:

[0040] A computer program is stored thereon, and when the computer program is executed by the memory, the steps of any one of the text dataset cleaning methods described in the first aspect are implemented.

[0041] Compared with the prior art, the technical solution provided by the embodiment of the present invention comprises the following steps: receiving a text dataset to be cleaned; classifying the text dataset to be cleaned into a first format text dataset and a second format text dataset based on a Bash script; performing a Chinese filtering operation on the first format text dataset and the second format text dataset to form a first format first text dataset and a second format first text dataset, respectively; converting the second format first text dataset into a first format second text dataset; performing encoding format conversion on the first format first text dataset and the first format second text dataset to form a first format third text dataset; and performing a data cleaning operation on the first format third text dataset, wherein the data cleaning operation includes: calculating the first format third text dataset using a message digest algorithm and performing a content deduplication operation. The present invention implements an efficient data classification and cleaning solution based on a Bash script and the Spark open source cluster computing framework, capable of processing TB-level Internet text datasets in a short time, thereby improving cleaning efficiency.

[0042] The technical solution provided by the embodiments of the present invention can use Bash scripts to classify internet text datasets and use the Spark open-source cluster computing framework as the computing framework to clean the text datasets. Since the Spark open-source cluster computing framework supports programming languages ​​such as Python, Java, and Scala, and Scala is a multi-paradigm programming language designed to integrate the characteristics of object-oriented programming and functional programming, offering high performance and simplicity, the Scala programming language boasts 10 times the performance of Python. Choosing Scala to write the data cleaning module improves cleaning performance.

[0043] At the same time, the technical solution provided by the embodiment of the present invention uses the BERT BASE model to perform text quality classification on the text dataset, further ensuring the high quality requirements of the text dataset. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0045] Figure 1 Flowchart of the text dataset cleaning method provided in Example 1 of the present invention;

[0046] Figure 2 This is a diagram of the text dataset cleaning architecture provided in Example 1 of the present invention;

[0047] Figure 3 A structural diagram of a text dataset cleaning device provided in Example 2 of the present invention;

[0048] Figure 4 The exemplary system provided for the third embodiment of the present invention can be used to implement various embodiments described in this application. DETAILED DESCRIPTION

[0049] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0050] It should be noted that, unless the context clearly requires otherwise, words such as “include” and “comprising” throughout the specification and claims should be interpreted as inclusive rather than exclusive or exhaustive; that is, as “including but not limited to”.

[0051] Furthermore, in the description of the present invention, unless otherwise specified, “plurality” means two or more.

[0052] Example 1

[0053] The embodiment of the present invention provides a text data set cleaning method, such as Figure 1 As shown, the method may include:

[0054] Step S1: receiving a text data set to be cleaned.

[0055] Specifically, the text dataset to be cleaned is downloaded and received from an electronic network disk or an e-book website. The text dataset to be cleaned is an Internet e-book.

[0056] The purpose of data cleansing can be understood as deleting or correcting data that is erroneous, incomplete, malformed, or redundant. Data cleansing not only corrects errors but also strengthens the consistency of data across various information systems. Specialized data cleansing software can automatically detect data files, correct errors, and integrate data into a consistent format across the enterprise.

[0057] Step S2: Classify the text data set to be cleaned into a first format text data set and a second format text data set based on a Bash script.

[0058] Specifically, since the text datasets (multiple e-books) downloaded from the Internet exist in various formats, such as TXT format, EPUB format, MOBI format, PDF format, WORD format, and AZW3 format, etc., and the final format to be obtained here is the TXT format.

[0059] First, the file formats were classified and counted. From the statistical results, it can be seen that the largest proportions are the TXT format and the EPUB format. Therefore, the downloaded e-book collection is classified into three formats here, namely the TXT format, the EPUB format, and other formats. The e-books in other formats are discarded here, and only the e-books in the TXT format and the EPUB format are retained.

[0060] Among them, the first-format text dataset here is an e-book in the TXT format; the second-format text dataset is an e-book in the EPUB format.

[0061] Step S3, perform a Chinese filtering operation on the first-format text dataset and the second-format text dataset to respectively form a first-format first text dataset and a second-format first text dataset.

[0062] Specifically, the text datasets downloaded from the Internet here, that is, the e-book catalogs, are nested complexly and only contain e-books in Chinese or English. Finally, only the Chinese e-books need to be retained, and the English e-books are deleted.

[0063] Step S3 further includes:

[0064] Step S31, determine whether the text data in the first-format text dataset is Chinese text data according to whether the full path of each text data in the first-format text dataset includes a first preset vocabulary. If so, delete it; if not, retain it; and / or determine whether the text data in the first-format text dataset is Chinese text data according to whether the first one hundred lines of each text data in the first-format text dataset include Chinese characters. If so, retain it; if not, delete it, to form a first-format first text dataset.

[0065] Specifically, here, two dimensions are used to determine whether the e-book in the TXT format is a Chinese e-book. The first dimension is to determine whether the e-book in the TXT format is a Chinese e-book from the full path of the e-book: Determine whether it is an English e-book according to whether the full path of the e-book contains the two vocabulary words "English" or "英语". If so, delete this e-book; if not, retain this e-book to form a first-format first text dataset.

[0066] Among them, the first preset vocabulary is vocabulary words such as "English" or "英语", and the "English" vocabulary is not case-sensitive.

[0067] The second dimension is to determine whether the first one hundred lines of the e-book file in TXT format include Chinese characters. If the first one hundred lines of the e-book file do not include Chinese characters, then delete this e-book; if the first one hundred lines of the e-book file include Chinese characters, then retain this e-book to form the first-format first text dataset.

[0068] Among them, the first-format first text dataset is an e-book in Chinese TXT format after Chinese filtering.

[0069] Step S32: Determine whether the text data in the second-format text dataset is Chinese text data according to whether the full path of each text data includes a first preset vocabulary. If so, delete it; if not, retain it to form the second-format first text dataset.

[0070] Specifically, here it is adopted to judge whether an EPUB-format e-book is a Chinese e-book from the full path of the e-book: judge whether it is an English e-book according to whether the full path of the e-book contains the two words "English" or "英语". If so, delete this e-book; if not, retain this e-book to form the second-format first text dataset.

[0071] Among them, the second-format first text dataset is an e-book in Chinese EPUB format after Chinese filtering.

[0072] Step S4: Convert the second-format first text dataset into the first-format second text dataset.

[0073] Specifically, here it correspondingly converts an e-book in Chinese EPUB format into an e-book in Chinese TXT format.

[0074] Among them, the first-format second text dataset is an e-book in Chinese TXT format converted from an e-book in Chinese EPUB format after Chinese filtering.

[0075] Step S5: Perform encoding format conversion on the first-format first text dataset and the first-format second text dataset to form the first-format third text dataset.

[0076] Specifically, the encoding formats of the e-books in Chinese TXT format can be divided into UTF-8 format, GB18030 format, GBK format, and GB2312 format. And the GB18030 format is compatible with the GBK format and the GB2312 format, so here it is set that the encoding formats of all the e-books in Chinese TXT format are divided into these two formats: UTF-8 and GB18030.

[0077] Here, the first format text dataset in Chinese is a Chinese e-book in TXT format; the second format text dataset in Chinese is a Chinese e-book in EPUB format; and the first format third text dataset is a Chinese e-book in UTF-8 format.

[0078] Understandably, after the above classification process, we obtain five categories of e-books: Chinese EPUB e-books, Chinese TXT e-books in UTF-8 encoding format, Chinese TXT e-books in GB18030 encoding format, English EPUB e-books, and English TXT e-books. Since only Chinese e-books are retained, the English EPUB e-books and English TXT e-books are deleted.

[0079] Step S5 further includes:

[0080] Step S51 : converting text data in the first format first text dataset and the first format second text dataset encoded in the second encoding format into text data in the first encoding format to form a first format third text dataset.

[0081] Specifically, the Chinese GB18030 format TXT e-book is converted into the Chinese UTF-8 format TXT e-book. The text data encoded in the first encoding format in the first format first text dataset and the first format second text dataset remains unchanged and is included in the first format third text dataset.

[0082] Among them, here, the text data encoded in the second encoding format in the first format first text data set and the first format second text data set is a TXT e-book in the Chinese GB18030 format; the text data encoded in the first encoding format in the first format first text data set and the first format second text data set is a TXT e-book in the Chinese UTF-8 format, that is, the first format third text data set.

[0083] Step S6: performing a data cleaning operation on the first format third text dataset, wherein the data cleaning operation includes: performing calculations on the first format third text dataset using a message digest algorithm and performing a content deduplication operation.

[0084] Specifically, since the text dataset downloaded from the Internet, i.e., multiple e-books, may contain duplicates, a message digest algorithm (MD5) is performed on the Chinese UTF-8 format TXT e-books, and the MD5 digest algorithm is used to perform content deduplication operations.

[0085] The Message-Digest Algorithm (MD5) is a widely used cryptographic hash function that generates a 128-bit (16-byte) hash value, ensuring the integrity and consistency of information transmission. MD5 replaces the MD4 algorithm. The algorithm's procedures are specified in RFC 1321.

[0086] Step S6 further includes:

[0087] Step S61, determining whether the number of Chinese characters in each text data in the first format third text data set is greater than or equal to a first threshold and whether the proportion of Chinese characters in the text data is greater than or equal to a second threshold;

[0088] If yes, keep the text data; if no, delete it.

[0089] Specifically, it is determined whether the number of Chinese characters in the Chinese UTF-8 encoded TXT e-book is greater than or equal to 150 characters, and whether the proportion of Chinese characters is greater than or equal to 60% to perform non-Chinese filtering. If the number of Chinese characters in the Chinese UTF-8 encoded TXT e-book is greater than or equal to 150 characters, and the proportion of Chinese characters is greater than or equal to 60%, then the e-book is retained; if the number of Chinese characters in the Chinese UTF-8 encoded TXT e-book is less than 150 characters, or the proportion of Chinese characters is less than 60%, then the e-book is deleted and non-Chinese or garbled characters are filtered out.

[0090] Here, the first threshold is 150, and the second threshold is 60%.

[0091] Step S62: Classify the collected sensitive words into first-category sensitive words and second-category sensitive words;

[0092] Traversing the first format third text data set;

[0093] If the text data in the first format third text data set includes the first category of sensitive words, deleting them;

[0094] If the text data in the first format third text data set includes the second category of sensitive words, the paragraph including the second category of sensitive words is deleted, and the rest of the text data is retained.

[0095] Specifically, the 9,759 sensitive words collected were classified. The first category of sensitive words includes pornographic sensitive words, and the second category of sensitive words includes political, violent, or advertising sensitive words. In terms of algorithm, the deterministic finite automaton (DFA) algorithm is used here to construct a sensitive word tree structure, and the Chinese UTF-8 encoded format TXT e-book is traversed to determine whether the e-book contains any category of sensitive words. If the e-book matches the first category of sensitive words, the e-book is deleted; if the e-book matches the second category of sensitive words, the paragraphs corresponding to the second category of sensitive words are deleted, and the rest of the e-book is retained.

[0096] Step S63: dividing the first format third text dataset into a first quality text dataset and a second quality text dataset using a BERT BASE model;

[0097] Delete the second quality text dataset.

[0098] Specifically, the transformer-based bidirectional encoder representation technology BERT (Bidirectional Encoder Representations from Transformers) is a pre-training technology for natural language processing (NLP) proposed by Google. The BERT model architecture is based on the Transformer and implements a multi-layer bidirectional Transformer encoder. The architecture includes two types of pre-training models, as shown in Table 1: (1) BERT BASE model, including a 12-layer, 768-dimensional, 12 self-attention heads, and a 110M parameter neural network structure; (2) BERT LARGE model, including a 24-layer, 1024-dimensional, 16 self-attention heads, and a 340M parameter neural network structure. The BERT BASE model is used here to complete the text quality classification of the Chinese UTF-8 encoded format TXT e-book.

[0099] Model Number of Transformer layers (L) Hiddenunits(H) self-attention heads (A) Total parameters BERT (base) 12 768 12 110 million BERT (large) 24 1024 16 340 million

[0100] Table 1

[0101] NLP (Natural Language Processing) is a subfield of artificial intelligence and linguistics. It explores how to process and utilize natural language. Natural language processing involves multiple steps, including cognition, comprehension, and generation.

[0102] It can be understood that through the BERT BASE model, the Transformer-based bidirectional encoder, the output layer corresponding to the first token is connected to the fully connected layer, which divides the Chinese UTF-8 encoded format TXT e-books into two categories: high-quality Chinese UTF-8 encoded format TXT e-books and low-quality Chinese UTF-8 encoded format TXT e-books. Ultimately, only high-quality Chinese UTF-8 encoded format TXT e-books are retained.

[0103] Here, the first quality text dataset is high-quality e-books, which are e-books with coherent sentences and clear content; the second quality text dataset is low-quality e-books, which are e-books with incoherent sentences and unclear content.

[0104] Step S64, removing special characters from each text data in the first format third text data set;

[0105] The traditional Chinese characters of each text data in the first format third text data set are converted into simplified Chinese characters.

[0106] Specifically, the JAVA programming language is used to implement a conversion program for converting traditional Chinese characters in the Chinese UTF-8 encoded format TXT e-book into simplified Chinese characters, which effectively improves the conversion performance;

[0107] Special characters in the Chinese TXT e-book in UTF-8 encoding format are removed, wherein the special characters include emoticons, garbled characters, invisible characters, etc.

[0108] Step S7, performing sampling verification on the cleaned first format third text data set;

[0109] The first format third text data set that passes the inspection is stored.

[0110] Specifically, after the text dataset classification, conversion, and cleaning steps, manual sampling verification is performed to ensure that the text dataset generated after the cleaning process is within the required test data range. After manual sampling verification by multiple people, the text dataset generated after the cleaning process is tested to see if it is within the required test data range.

[0111] In order to improve performance, a distributed database is not used for storage here. Instead, local file storage on the server is used to store the cleaned high-quality Chinese UTF-8 encoded TXT e-book data.

[0112] It is understandable that if Figure 2The following diagram shows an efficient internet text dataset cleaning architecture design based on Bash scripts and the Spark computing framework; it includes: a data download module, a data classification and conversion module, a data cleaning module, a verification module, and a data storage module. The data classification and conversion module is based on a Bash script; the data cleaning module is based on the Spark computing framework; after data cleaning, the cleaned high-quality Chinese e-book data is stored using local server file storage. Here, Bash scripts are used to classify internet e-books, and the Spark framework is used as the computing framework for data cleaning. This allows for processing terabyte-level internet text datasets in a short period of time, improving cleaning efficiency. Data cleaning will be integrated with deep learning in the future to improve text dataset cleaning methods, thereby more accurately obtaining high-quality Chinese e-book datasets.

[0113] The text dataset cleaning method provided in the embodiment of the present invention can be further improved and optimized without departing from the technical solution of the present invention, and these improvements and optimizations should also be considered as the scope of protection of the present invention.

[0114] An embodiment of the present invention provides a method for cleaning a text dataset. The technical solution provided by the embodiment of the present invention comprises the following steps: receiving a text dataset to be cleaned; classifying the text dataset to be cleaned into a first format text dataset and a second format text dataset based on a Bash script; performing a Chinese filtering operation on the first format text dataset and the second format text dataset to form a first format first text dataset and a second format first text dataset, respectively; converting the second format first text dataset into a first format second text dataset; performing encoding format conversion on the first format first text dataset and the first format second text dataset to form a first format third text dataset; and performing a data cleaning operation on the first format third text dataset, wherein the data cleaning operation includes calculating the first format third text dataset using a message digest algorithm and performing a content deduplication operation. The present invention implements an efficient data classification and cleaning solution based on a Bash script and the Spark open source cluster computing framework, capable of processing TB-level Internet text datasets in a short period of time, thereby improving cleaning efficiency.

[0115] Example 2

[0116] The present invention provides a text data set cleaning device, such as Figure 3 As shown, it includes a memory, a processor, and a computer program stored in the memory and capable of running on the processor. When the processor executes the computer program, the text data set cleaning method can be implemented. The processor includes a receiving module, a processing module, a detection module, and a storage module.

[0117] In this embodiment, the receiving module is used to receive a text data set to be cleaned; the processing module is used to classify the text data set to be cleaned into a first format text data set and a second format text data set based on a Bash script; the detection module is used to perform sampling verification on the cleaned first format and third format text data sets; and the storage module is used to store the first format and third format text data sets that have passed the inspection.

[0118] Furthermore, the processing module is configured to perform a Chinese filtering operation on the first format text dataset and the second format text dataset to form a first format first text dataset and a second format first text dataset respectively.

[0119] Furthermore, the processing module may be further configured to convert the second-format first text dataset into the first-format second text dataset.

[0120] Furthermore, the processing module may be configured to convert the encoding format of the first format first text dataset and the first format second text dataset to form a first format third text dataset.

[0121] Furthermore, the processing module is used to perform a data cleaning operation on the first format third text data set, wherein the data cleaning operation includes: calculating the first format third text data set through a message digest algorithm and performing a content deduplication operation.

[0122] Furthermore, the processing module may be further configured to determine whether the text data in the first format text data set is Chinese text data based on whether the full path of each text data set includes a first preset vocabulary, and if so, delete the text data; if not, retain the text data; and / or determine whether the text data is Chinese text data based on whether the first 100 lines of each text data set in the first format text data set include Chinese characters, and if so, retain the text data; if not, delete the text data, to form a first format first text data set;

[0123] Whether the text data is Chinese text data is determined based on whether the full path of each text data in the second format text data set includes the first preset vocabulary. If so, it is deleted; if not, it is retained to form the second format first text data set.

[0124] Furthermore, the processing module is configured to convert text data in the first format first text dataset and the first format second text dataset whose encoding format is the second encoding format into text data in the first encoding format to form a first format third text dataset.

[0125] Furthermore, the processing module may be further configured to determine whether the number of Chinese characters in each text data in the first format third text data set is greater than or equal to a first threshold and whether the proportion of Chinese characters in the text data is greater than or equal to a second threshold;

[0126] If yes, keep the text data; if no, delete it.

[0127] Furthermore, the processing module may be further configured to classify the collected sensitive words into first-category sensitive words and second-category sensitive words;

[0128] Traversing the first format third text data set;

[0129] If the text data in the first format third text data set includes the first category of sensitive words, deleting them;

[0130] If the text data in the first format third text data set includes the second category of sensitive words, the paragraph including the second category of sensitive words is deleted, and the rest of the text data is retained.

[0131] Further, the processing module may be used to separate the first format third text dataset into a first quality text dataset and a second quality text dataset through a BERT BASE model;

[0132] Delete the second quality text dataset.

[0133] Furthermore, the processing module may be used to remove special characters from each text data in the first format third text data set;

[0134] The traditional Chinese characters of each text data in the first format third text data set are converted into simplified Chinese characters.

[0135] The technical solution provided by the embodiment of the present invention has the following beneficial effects:

[0136] The technical solution provided by an embodiment of the present invention comprises the following steps: receiving a text dataset to be cleaned; classifying the text dataset to be cleaned into a first format text dataset and a second format text dataset based on a Bash script; performing a Chinese filtering operation on the first format text dataset and the second format text dataset to form a first format first text dataset and a second format first text dataset, respectively; converting the second format first text dataset into a first format second text dataset; performing encoding format conversion on the first format first text dataset and the first format second text dataset to form a first format third text dataset; and performing a data cleaning operation on the first format third text dataset, wherein the data cleaning operation includes: calculating the first format third text dataset using a message digest algorithm and performing a content deduplication operation. The present invention implements an efficient data classification and cleaning solution based on a Bash script and the Spark open source cluster computing framework, capable of processing TB-level Internet text datasets in a short time, thereby improving cleaning efficiency.

[0137] Example 3

[0138] The present invention provides a computer storage medium comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the following steps are implemented:

[0139] Receive the text dataset to be cleaned;

[0140] Classifying the text data set to be cleaned into a first format text data set and a second format text data set based on a Bash script;

[0141] Performing a Chinese filtering operation on the first format text dataset and the second format text dataset to form a first format first text dataset and a second format first text dataset, respectively;

[0142] converting the second-format first text dataset into the first-format second text dataset;

[0143] Converting the first text dataset in the first format and the second text dataset in the first format into encoding formats to form a third text dataset in the first format;

[0144] A data cleaning operation is performed on the first format third text data set, wherein the data cleaning operation includes: calculating the first format third text data set using a message digest algorithm and performing a content deduplication operation.

[0145] Furthermore, performing Chinese filtering operations on the first format text dataset and the second format text dataset to form the first format first text dataset and the second format first text dataset respectively includes:

[0146] determining whether the text data in the first format text data set is Chinese text data based on whether the full path of each text data set includes a first preset vocabulary, and if so, deleting the text data; if not, retaining the text data; and / or determining whether the text data is Chinese text data based on whether the first 100 lines of each text data set in the first format text data set include Chinese characters, and if so, retaining the text data; if not, deleting the text data, to form a first format first text data set;

[0147] Whether the text data is Chinese text data is determined based on whether the full path of each text data in the second format text data set includes the first preset vocabulary. If so, it is deleted; if not, it is retained to form the second format first text data set.

[0148] Furthermore, the converting the encoding format of the first format first text dataset and the first format second text dataset to form the first format third text dataset includes:

[0149] The text data in the first format first text data set and the first format second text data set, whose encoding format is the second encoding format, are converted into text data in the first encoding format to form a first format third text data set.

[0150] Furthermore, before calculating the first format third text data set using a message digest algorithm and performing content deduplication operations, the method further includes:

[0151] The first format third text dataset is divided into a first quality text dataset and a second quality text dataset by using a BERT BASE model;

[0152] Delete the second quality text dataset.

[0153] Furthermore, the performing of data cleaning operation on the first format third text dataset further includes:

[0154] Determine whether the number of Chinese characters in each text data in the first format third text data set is greater than or equal to a first threshold and whether the proportion of Chinese characters in the text data is greater than or equal to a second threshold;

[0155] If yes, keep the text data; if no, delete it.

[0156] Furthermore, the performing of data cleaning operation on the first format third text dataset further includes:

[0157] The collected sensitive words are divided into the first category of sensitive words and the second category of sensitive words;

[0158] Traversing the first format third text data set;

[0159] If the text data in the first format third text data set includes the first category of sensitive words, deleting them;

[0160] If the text data in the first format third text data set includes the second category of sensitive words, the paragraph including the second category of sensitive words is deleted, and the rest of the text data is retained.

[0161] Furthermore, the performing of data cleaning operation on the first format third text dataset further includes:

[0162] removing special characters from each text data in the first format third text data set;

[0163] The traditional Chinese characters of each text data in the first format third text data set are converted into simplified Chinese characters.

[0164] Furthermore, the text dataset cleaning method includes:

[0165] Performing sampling verification on the cleaned first format third text data set;

[0166] The first format third text data set that passes the inspection is stored.

[0167] Figure 4 The exemplary system provided in the third embodiment of the present invention can be used to implement various embodiments described in this application;

[0168] like Figure 4 As shown, in some embodiments, the system can be used as any of the above-mentioned devices for cleaning text datasets in each of the above-mentioned embodiments. In some embodiments, the system may include one or more computer-readable media (e.g., system memory or NVM / storage device) having results and one or more processors (e.g., (one or more) processors) coupled to the one or more computer-readable media and configured to execute the results to implement the module to perform the actions described in this application.

[0169] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by computer programs and the corresponding hardware. The above-mentioned computer programs can be stored in a non-volatile computer-readable storage medium. When the computer programs are executed, they can include the processes of the above-mentioned embodiments. Among them, any reference to memory, storage, database or other media used in the embodiments provided by the present invention can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0170] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0171] The above-described embodiments merely illustrate several implementations of the present invention, and while their descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the patent. It should be noted that a person skilled in the art would be able to make numerous variations and improvements without departing from the spirit of the present invention, all of which fall within the scope of protection of the present invention. Therefore, the scope of protection of the patent for this invention shall be determined by the appended claims.

Claims

1. A text data set cleaning method, characterized in that: The method comprises: Receive the text dataset to be cleaned; Classifying the text data set to be cleaned into a first format text data set and a second format text data set based on a Bash script; Performing a Chinese filtering operation on the first format text dataset and the second format text dataset to form a first format first text dataset and a second format first text dataset, respectively; converting the second-format first text dataset into the first-format second text dataset; Converting the first text dataset in the first format and the second text dataset in the first format into encoding formats to form a third text dataset in the first format; Performing a data cleaning operation on the first format third text data set, wherein the data cleaning operation includes: calculating the first format third text data set using a message digest algorithm and performing a content deduplication operation; The performing Chinese filtering operations on the first format text dataset and the second format text dataset to respectively form the first format first text dataset and the second format first text dataset includes: determining whether the text data in the first format text data set is Chinese text data based on whether the full path of each text data set includes a first preset vocabulary, and if so, deleting the text data; if not, retaining the text data; and / or determining whether the text data is Chinese text data based on whether the first 100 lines of each text data set in the first format text data set include Chinese characters, and if so, retaining the text data; if not, deleting the text data, to form a first format first text data set; determining whether the text data is Chinese text data based on whether the full path of each text data in the second format text data set includes the first preset vocabulary, and if so, deleting it; if not, retaining it, to form the second format first text data set; Among them, the first format text dataset is an e-book in TXT format; the second format text dataset is an e-book in EPUB format; and the first format third text dataset is a TXT e-book in Chinese UTF-8 format.

2. The text data set cleaning method according to claim 1, characterized in that: The step of converting the encoding format of the first-format first text dataset and the first-format second text dataset to form a first-format third text dataset includes: The text data in the first format first text data set and the first format second text data set whose encoding format is the second encoding format are converted into text data in the first encoding format to form a first format third text data set.

3. The text data set cleaning method according to claim 1, characterized in that: Before calculating the first format third text data set using a message digest algorithm and performing content deduplication operations, the method further includes: The first format third text dataset is divided into a first quality text dataset and a second quality text dataset by using a BERT BASE model; Delete the second quality text dataset.

4. The text data set cleaning method according to claim 3, characterized in that: The data cleaning operation on the first format third text data set further includes: Determine whether the number of Chinese characters in each text data in the first format third text data set is greater than or equal to a first threshold and whether the proportion of Chinese characters in the text data is greater than or equal to a second threshold; If yes, keep the text data; if no, delete it.

5. The text data set cleaning method according to claim 3, characterized in that: The data cleaning operation on the first format third text data set further includes: The collected sensitive words are divided into the first category of sensitive words and the second category of sensitive words; Traversing the first format third text data set; If the text data in the first format third text data set includes the first category of sensitive words, deleting them; If the text data in the first format third text data set includes the second category of sensitive words, the paragraph including the second category of sensitive words is deleted, and the rest of the text data is retained.

6. The text data set cleaning method according to claim 3, characterized in that: The data cleaning operation on the first format third text data set further includes: removing special characters from each text data in the first format third text data set; The traditional Chinese characters of each text data in the first format third text data set are converted into simplified Chinese characters.

7. The text data set cleaning method according to claim 1, characterized in that: The method comprises: Performing sampling verification on the cleaned first format third text data set; The first format third text data set that passes the inspection is stored.

8. A text data set cleaning device, comprising a memory, a processor, and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the steps of the method according to any one of claims 1 to 7.

9. A computer storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a memory, the steps of the method according to any one of claims 1 to 7 are implemented.

Images