A corpus cleaning method, device, equipment and storage medium

Abstract

The application provides a corpus cleaning method, device and equipment and a storage medium. On one hand, the parallel processing capability of a graphics processor can be reasonably applied, and the deduplication processing operation in the corpus cleaning process is migrated to the graphics processor for parallel execution, so that the deduplication processing efficiency is effectively improved. On the other hand, the confidence calibration and confidence filtering are performed on the document classification result output by the document classification model, so that the documents with a confidence level lower than a preset standard can be effectively removed in the data cleaning process, the corpus quality of the target corpus obtained after cleaning is improved, and the efficiency and quality of the corpus cleaning can be considered.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and more specifically, to a corpus cleaning method, apparatus, device, and storage medium. Background Technology

[0002] In large-scale corpus cleaning, the industry commonly adopts a distributed streaming processing architecture based on CPU (Central Processing Unit) clusters. Examples include cleaning pipelines built on Apache Spark (a distributed computing engine) or Flink (an open-source streaming computing engine), or open-source frameworks such as DataJuicer. In a typical processing flow, terabytes (TB) of raw corpus data are usually divided into multiple batches, and then processed sequentially in a CPU server cluster, including basic filtering, corpus identification, approximate deduplication, and text classification and quality assessment.

[0003] Specifically, in the basic filtering stage, existing technologies rely on the CPU to perform string length statistics and rule-based regular expression matching to remove webpage snapshots that are too short or contain a large number of HTML (HyperText Markup Language) tags. In the corpus identification stage, existing technologies often use lightweight machine learning models such as fastText to perform language recognition on the CPU. In the approximate deduplication stage, existing technologies typically use the MinHash algorithm, where the CPU extracts words from each document, calculates the hash values ​​of multiple hash functions one by one, and generates a document signature by comparing the minimum value in a loop. Duplicate documents are then found using Locality Sensitive Hashing (LSH) bucketing. In the text classification stage, existing technologies directly predict the document category of each document through the model, thereby completing the category division of the corpus. After completing the above processing stages, the cleaned corpus is finally obtained as training data for subsequent training of large language models.

[0004] However, the aforementioned existing technical solutions have significant technical shortcomings in the approximate deduplication and text classification stages. In the approximate deduplication stage, the existing technology relies solely on the CPU to serially extract words from each document, calculate the hash values ​​of multiple hash functions one by one, and then generate a document signature by comparing the minimum value in a loop. This results in low deduplication efficiency. In the text classification stage, the existing technology directly uses the document classification results output by the model without verifying the reliability of the model's output. This leads to low accuracy in the document classification results, which can easily cause misclassification of the corpus and affect the training results of subsequent large language models. Summary of the Invention

[0005] In view of this, this application provides a corpus cleaning method, apparatus, device, and storage medium. On the one hand, it can rationally utilize the parallel processing capabilities of the graphics processor, migrating the deduplication operation in the corpus cleaning process to the graphics processor for parallel execution, effectively improving the deduplication efficiency. On the other hand, by performing confidence calibration and confidence filtering on the document classification results output by the document classification model, it can effectively remove documents with confidence categories lower than the preset standard during the data cleaning process, which is conducive to improving the corpus quality of the target corpus obtained after cleaning. Thus, this application can balance the efficiency and quality of corpus cleaning.

[0006] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings.

[0007] In a first aspect, embodiments of this application provide a corpus cleaning method, the corpus cleaning method comprising: By performing a pre-filtering operation included in the coarse screening stage on the original corpus, texts that do not meet the preset quality conditions are filtered out from the original corpus, and a first intermediate corpus that meets the preset quality conditions is obtained. All documents in the first intermediate corpus are mapped to the video memory of the graphics processor. The hash values ​​of all words in all documents are calculated in parallel using multiple preset hash functions through a single kernel call of the graphics processor. The hash values ​​are then reduced to minimum values ​​along the word dimension in the video memory of the graphics processor to obtain the target signature matrix corresponding to the first intermediate corpus. Each row in the target signature matrix represents the target hash signature of each document in the first intermediate corpus. In the graphics processor's video memory, the target signature matrix is ​​subjected to local sensitive hashing bucketing to determine the duplicate documents contained in the first intermediate corpus; and based on the duplicate documents, the first intermediate corpus is deduplicated to obtain the deduplicated first intermediate corpus as the second intermediate corpus. The second intermediate corpus is input into a pre-trained document classification model, and the document classification model predicts the document category of each document in the second intermediate corpus to obtain the original classification logical value corresponding to each document in the second intermediate corpus; wherein, the original classification logical value is a vector composed of the unnormalized score output by the document classification model and corresponding to each preset document category; In the video memory of the graphics processor, the original classification logic value is calibrated with confidence to obtain the confidence calibration result corresponding to each document in the second intermediate corpus. Based on the confidence calibration result and the preset multi-level confidence threshold, the confidence category to which each document in the second intermediate corpus belongs is determined. From the second intermediate corpus, documents with confidence levels below a preset standard are filtered out, and the remaining documents are used as target corpus for training a large language model.

[0008] Secondly, embodiments of this application provide a corpus cleaning apparatus, the corpus cleaning apparatus comprising: The coarse screening module is used to filter out texts that do not meet the preset quality conditions from the original corpus by performing a preset pre-filtering operation included in the coarse screening stage, and obtain a first intermediate corpus that meets the preset quality conditions. The deduplication module maps all documents in the first intermediate corpus to the video memory of the graphics processor. It then uses a single kernel call of the graphics processor to calculate the hash values ​​of all terms in all documents using multiple preset hash functions in parallel. Finally, it performs minimum reduction on the hash values ​​along the term dimension in the video memory of the graphics processor to obtain the target signature matrix corresponding to the first intermediate corpus. Each row in the target signature matrix represents the target hash signature of each document in the entire corpus. In the graphics processor's video memory, the target signature matrix is ​​subjected to local sensitive hashing bucketing to determine the duplicate documents contained in the first intermediate corpus; and based on the duplicate documents, the first intermediate corpus is deduplicated to obtain the deduplicated first intermediate corpus as the second intermediate corpus. The classification module is used to input the second intermediate corpus into a pre-trained document classification model, predict the document category of each document in the second intermediate corpus through the document classification model, and obtain the original classification logical value corresponding to each document in the second intermediate corpus; wherein, the original classification logical value is a vector composed of the unnormalized score output by the document classification model and the corresponding preset document category; The confidence filtering module is used to perform confidence calibration on the original classification logic value in the graphics processor's video memory to obtain the confidence calibration result corresponding to each document in the second intermediate corpus, and determine the confidence category to which each document in the second intermediate corpus belongs based on the confidence calibration result and the preset multi-level confidence threshold. From the second intermediate corpus, documents with confidence levels below a preset standard are filtered out, and the remaining documents are used as target corpus for training a large language model.

[0009] Thirdly, embodiments of this application provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the corpus cleaning method described above.

[0010] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program, which, when run by a processor, performs the steps of the corpus cleaning method described above.

[0011] The technical solutions provided by the embodiments of this application may include the following beneficial effects: The corpus cleaning method, apparatus, device, and storage medium provided in this application have the following advantages: First, they can rationally utilize the parallel processing capabilities of graphics processors, migrating the deduplication operation in the corpus cleaning process to the graphics processor for parallel execution, thus effectively improving the deduplication efficiency. Second, by performing confidence calibration and confidence filtering on the document classification results output by the document classification model, documents with confidence categories lower than the preset standard can be effectively removed during the data cleaning process, which is beneficial to improving the quality of the target corpus obtained after cleaning. Thus, this application can balance the efficiency and quality of corpus cleaning. Attached Figure Description

[0012] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 A flowchart illustrating a corpus cleaning method provided in an embodiment of this application is shown. Figure 2 This paper shows a schematic diagram of the structure of a corpus cleaning apparatus provided in an embodiment of this application; Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0014] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the accompanying drawings in this application are for illustrative and descriptive purposes only and are not intended to limit the scope of protection of this application. Furthermore, it should be understood that the schematic drawings are not drawn to scale. The flowcharts used in this application illustrate operations implemented according to some embodiments of this application. It should be understood that the operations in the flowcharts may not be implemented in sequence, and steps without logical contextual relationships may be reversed or implemented simultaneously. In addition, those skilled in the art, guided by the content of this application, may add one or more other operations to the flowcharts, or remove one or more operations from the flowcharts.

[0015] Furthermore, the described embodiments are merely some, not all, of the embodiments of this application. The components of the embodiments of this application described and illustrated herein can typically be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0016] It should be noted that the term "comprising" will be used in the embodiments of this application to indicate the presence of the features declared thereafter, but does not exclude the addition of other features.

[0017] In one embodiment of this application, a corpus cleaning method can be run on an electronic device; wherein the electronic device can be a terminal device or a server; the terminal device can be a local terminal device. When the corpus cleaning method runs on a server, the corpus cleaning method can be implemented and executed based on a cloud interactive system, wherein the cloud interactive system includes a server and client devices (i.e., terminal devices).

[0018] To facilitate understanding of the embodiments of this application, a detailed description of a corpus cleaning method, apparatus, device, and storage medium provided in the embodiments of this application is provided below.

[0019] Reference Figure 1 As shown, Figure 1 The diagram illustrates a flowchart of a corpus cleaning method provided in an embodiment of this application, wherein the corpus cleaning method includes steps S101-S106; specifically: S101, by performing a preset pre-filtering operation included in the coarse screening stage on the original corpus, texts that do not meet the preset quality conditions are filtered out from the original corpus, and a first intermediate corpus that meets the preset quality conditions is obtained.

[0020] Here, the original corpus refers to the text data used as training data for the large language model; the original corpus contains multiple documents, and each document contains multiple texts. The specific content of the original corpus can be determined according to the actual model training requirements of the large language model, and this embodiment of the application does not impose any limitations on this.

[0021] In this embodiment, the corpus cleaning process for the original corpus can be divided into two stages: lightweight operator pre-processing and heavy operator post-processing. The lightweight operator pre-processing stage is also the coarse screening stage to which step S101 belongs (equivalent to step S101 being the corpus cleaning step within the lightweight operator pre-processing stage). In the lightweight operator pre-processing stage, the aforementioned preset pre-filtering operation consumes relatively little computing resources (i.e., significantly less computing resources than the deduplication process shown in subsequent steps S102-S103 and the document classification and confidence filtering process shown in subsequent steps S104-S106). Therefore, the above-mentioned preset pre-filtering operation (i.e., step S101 is executed in the CPU) can be performed on the original corpus input into the CPU of the electronic device. In this way, the above-mentioned preset pre-filtering operation, which consumes little computing resources and has low computational cost, can filter out obviously low-quality corpus (i.e., corpus that does not meet the preset quality conditions) that account for a large proportion of the original corpus. This helps to reduce the amount of data processing in subsequent time-consuming steps (i.e., steps S102-S106, which belong to the post-stage of the weight operator). Therefore, the efficiency of corpus cleaning can be significantly improved compared with the prior art.

[0022] It should be noted that the aforementioned preset quality conditions can be a set of rules for quantitatively evaluating the quality of the corpus from multiple different dimensions. The rules within this set can be used to determine whether each text in the original corpus meets the minimum admission criteria for entering the next cleaning stage (i.e., the post-weighting operator stage), using a single text as the smallest judgment unit. In this embodiment, the preset quality conditions can cover filtering criteria across multiple dimensions, such as text form compliance, text content integrity, and text semantic rationality. This embodiment does not impose any mandatory limitations on the specific content of the aforementioned preset quality conditions (which is equivalent to the specific rules included in the aforementioned set of rules).

[0023] In this embodiment, the aforementioned lightweight operator pre-processing stage can be divided into four sub-stages according to the cleaning order of the original corpus: "fast interception stage," "text standardization stage," "deep quality filtering stage," and "confusion filtering stage." Within the lightweight operator pre-processing stage, these four sub-stages can be executed sequentially in order of increasing computational resource consumption. This allows the earlier sub-stages to filter out some unqualified text (i.e., text that does not meet preset quality conditions), reducing the data processing volume of subsequent sub-stages.

[0024] Here, since each sub-stage can contain one or more preset pre-filtering operations, the preset pre-filtering operations involved in each sub-stage will be explained in detail below according to the execution order of the four sub-stages mentioned above: 1. Fast Interception Stage: The fast interception stage is located at the beginning of the coarse screening stage. Its purpose is to quickly identify and discard individual texts in the original corpus that clearly lack semantic value or have abnormal formatting with extremely low computational overhead. In the embodiments of this application, the fast interception stage may include two sub-steps: text length filtering and language recognition filtering.

[0025] Specifically, in the text length filtering sub-step, the preset pre-filtering operation that can be performed is to "filter out single texts whose character count is outside the preset character count range"; where, within the CPU, the electronic device can directly count the number of Unicode (Unified Code, a standard character encoding standard) characters of each text in the original corpus. If the number of Unicode characters of a text is outside the preset character count range (e.g., [10, 12000]), then the text can be discarded (equivalent to determining that the text does not meet the preset quality conditions).

[0026] It should be noted that text with too few characters usually lacks complete semantic information and cannot provide effective training signals for large language models; while text with too many characters is often caused by web crawlers incorrectly splicing content from multiple pages or by abnormal data formats, and is also unsuitable as training data for large language models. Based on this, when setting the above-mentioned preset character count range, the minimum number of characters with complete semantic information can be used as the lower limit of the preset character count range, and the maximum number of characters without abnormalities can be used as the upper limit of the preset character count range.

[0027] Specifically, in the language recognition filtering sub-step, the preset pre-filtering operation that can be performed is to "filter out single texts whose language recognition results do not meet the preset recognition conditions"; wherein, the language recognition result is the probability value of a single text being input into the language recognition model and output by the language recognition model as belonging to the target language; and the target language can be determined according to the model training requirements of the large language model (for example, if you want to train a large language model for processing Chinese corpora, you can set the target language to Chinese).

[0028] For example, a lightweight language recognition model such as fastText (e.g., the lid.176.bin model for multi-classification based on character n-gram features) can be used to calculate the probability that each text in the original corpus belongs to the target language (e.g., Chinese), thus obtaining the probability value of each text belonging to the target language. If the probability value of a text belonging to the target language is lower than a preset probability threshold (e.g., lower than 0.35), it can be determined that the language recognition result of that text does not meet the preset recognition conditions (equivalent to the preset recognition conditions being that the probability value of belonging to the target language is greater than or equal to the preset probability threshold).

[0029] It should be noted that after the above-mentioned rapid interception stage of cleaning, a large number of obviously unqualified texts in the original corpus can be filtered out (i.e., texts whose character count is outside the preset character count range, and texts whose language recognition results do not meet the preset recognition conditions).

[0030] 2. Text Standardization Stage: After the fast interception is completed, the intercepted text (i.e., the remaining text after the preset pre-filtering operation in the fast interception stage) can be processed to standardize the text representation and reduce noise interference in subsequent deep quality filtering and deduplication.

[0031] It should be noted that the pre-filtering operation performed in the text standardization stage is different from the pre-filtering operation performed in the aforementioned fast interception stage. Its purpose is not to filter out unqualified text, but to standardize and repair the text content in order to reduce the noise generated in the subsequent cleaning process.

[0032] Specifically, the pre-filtering operation performed during the text standardization phase is "text standardization processing operation to convert a single text into the target text format"; this text standardization processing operation may include, but is not limited to: Traditional / Simplified Chinese conversion (using OpenCC tools to convert Traditional Chinese characters in the text to their corresponding Simplified Chinese characters), HTML tag cleaning (using a DOM parser to remove all tags, script blocks, and style blocks contained in the text, and escaping and repairing any remaining HTML entities; simultaneously, using non-backtracking regular expression engines such as RE2 to match and remove email addresses, IP addresses, hyperlinks, and other elements from the text). Meaningless templated content such as copyright notices), Unicode encoding repair (the ftfy library can be used to handle garbled characters and non-standard control characters caused by encoding conversion errors), removal of repeated sentences in documents (the text is divided into sentences according to punctuation marks, a hash table is built for all sentences with a length of no less than 2 characters, and repeated sentences in the same document are deleted, thereby effectively removing repeated paragraphs and navigation menu content introduced by crawlers), and standardization of whitespace characters (all non-standard whitespace characters such as full-width spaces and zero-width spaces are uniformly replaced with standard spaces, and multiple consecutive whitespace characters are merged into one).

[0033] 3. Deep Quality Filtering Stage: After completing the text standardization process, deep quality filtering can be performed on the standardized text. In this stage, multiple lightweight indicators based on text statistical features can be used to filter the text content, identifying and discarding low-quality text with obvious anomalies in its content composition.

[0034] Here, the preset pre-filtering operation performed in the deep quality filtering stage is to "filter out single texts whose text statistical features do not meet the preset feature conditions"; wherein, the text statistical features are determined based on at least one preset text statistical feature index (i.e., the lightweight index based on the above text statistical features).

[0035] Specifically, the aforementioned text statistical feature indicators may include at least one of the following: (1) Alphanumeric ratio filtering: Calculate the ratio of the number of letters, numbers and Chinese characters in a single text to the total number of characters in the text. If the ratio exceeds the preset ratio threshold (e.g., 0.9), it indicates that the text is almost entirely composed of letters and numbers, lacks punctuation marks and natural language structure, and usually belongs to non-natural language content such as encoded sequences, keys or Base64 encoding, and should be discarded.

[0036] (2) Average line length and maximum line length: Divide the text into lines according to the newline character and calculate the average line length and maximum line length respectively. If the average line length is lower than the preset line length lower limit (e.g., 10 characters), it means that the text content is seriously fragmented. If the maximum line length exceeds the preset line length upper limit (e.g., 10,000 characters), it often means that binary content is incorrectly embedded in the text. Both of the above abnormal situations should be discarded.

[0037] (3) Character repetition ratio: Use a sliding window to take character n-grams with a length of a preset window size (e.g., 10), and count the proportion of the number of repeated n-grams to the total number of n-grams. If the proportion exceeds the preset repetition ratio threshold (e.g., 0.5), it can be determined that the text contains a large amount of circularly filled garbage content and should be discarded.

[0038] (4) Sensitive word ratio: Load the preset Chinese sensitive word dictionary and count the proportion of sensitive word characters to the total number of characters in the text. If the proportion exceeds the preset sensitive word ratio threshold (e.g., 0.0045), it indicates that the text may contain a large amount of low-quality and illegal content and should be discarded.

[0039] 4. Perplexity Filtering Stage: As the last sub-stage of the coarse screening stage (i.e., the lightweight operator pre-screening stage), the purpose of the perplexity filtering stage is to further screen the corpus from the dimension of the semantic rationality of the text.

[0040] Specifically, the pre-filtering operation performed in the perplexity filtering stage is to "filter out single texts whose perplexity calculation results are higher than the preset perplexity threshold". In this process, a pre-trained statistical language model can be used to calculate the perplexity of each text to obtain the perplexity value of each text. If the calculated perplexity value is higher than the preset perplexity threshold, it can be determined that the text deviates significantly from the target language (such as Chinese) in terms of statistical regularity (e.g., the text may contain a large amount of garbled characters or noise content that does not conform to the natural language grammar structure), and should be discarded.

[0041] For example, KenLM, an n-gram statistical language model, can be used as the pre-trained statistical language model to calculate the perplexity of each text during the perplexity filtering stage. Internally, the KenLM model uses a Bloom filter to compress and store the n-gram counts in the text, resulting in a query complexity of approximately O(n). This represents a speed advantage of two to three orders of magnitude compared to neural network language models, making it more suitable for batch processing of terabyte-scale corpora on a CPU.

[0042] It should be noted that PPL (Perplexity) is a classic metric in the field of natural language processing for measuring the predictive ability of a language model. It is used to quantify the "predictive uncertainty" of the model on test data. In this embodiment, the lower the perplexity of the text, the more the text conforms to the statistical laws of the target language (such as Chinese). Therefore, in the perplexity filtering stage, it is necessary to filter out single texts with high perplexity (i.e., higher than the preset perplexity threshold).

[0043] It should be noted that the specific pre-filtering operations included in each of the above sub-stages, as well as the execution order of each pre-filtering operation, can be flexibly selected and adjusted by those skilled in the art based on the characteristics of the actual corpus data and the cleaning requirements. For example, in some application scenarios, only one or more pre-filtering operations can be selected and combined; or, for example, the threshold parameters involved in each pre-filtering operation can be appropriately adjusted according to the actual quality of the original corpus.

[0044] Based on this, the aforementioned preset pre-filtering operations include at least one of the following: filtering out single text entries whose character count falls outside a preset character count range; filtering out single text entries whose language recognition results do not meet preset recognition conditions; text standardization processing operations that convert single text entries into target text formats; filtering out single text entries whose text statistical features do not meet preset feature conditions; and filtering out single text entries whose perplexity calculation results are higher than a preset perplexity threshold. This application embodiment does not impose any mandatory limitations on the specific operation types and quantities included in the aforementioned preset pre-filtering operations.

[0045] S102, map all documents in the first intermediate corpus to the graphics processor's video memory, calculate the hash values ​​of all words in all documents under multiple preset hash functions through a single kernel call of the graphics processor, and perform minimum value reduction on the hash value calculation results along the word dimension in the graphics processor's video memory to obtain the target signature matrix corresponding to the first intermediate corpus.

[0046] Here, step S102 belongs to the deduplication process in the post-stage of the weight operator mentioned above. After obtaining the first intermediate corpus, considering that the first intermediate corpus may contain a large number of documents with repeated or highly similar content, this embodiment of the application adopts an approximate deduplication scheme based on the MinHash algorithm to quickly estimate the similarity between documents in the first intermediate corpus, so as to find and remove duplicate documents whose similarity exceeds a preset threshold.

[0047] It should be noted that, unlike the existing technology that performs MinHash hash calculations serially on the CPU, the embodiments of this application completely migrate the triple nested loop calculation of "document-term-hash function" in the MinHash algorithm to the GPU (Graphics Processing Unit) for execution. By mapping the above triple nested loop to a large-scale parallel thread grid on the graphics processor, the fully parallel calculation of the hash values ​​of all terms in the first intermediate corpus under multiple preset hash functions is realized, thereby completely eliminating the performance bottleneck caused by the serial calculation of the CPU in the existing technology.

[0048] Specifically, step S102 can be further divided into a hash value parallel computation process and a signature matrix reduction generation process; wherein, as an optional embodiment, the hash value parallel computation process in step S102 can be executed according to the method shown in steps a1-a2 below: Step a1: Concatenate the bytes of all documents into a continuous byte sequence, and map the continuous byte sequence and the starting offset of each document in the continuous byte sequence to the video memory of the graphics processor.

[0049] In this embodiment of the application, before performing the parallel hash value calculation step in the GPU, it is necessary to first complete the data preprocessing of all documents in the first intermediate corpus in the CPU.

[0050] Specifically, the raw bytes (e.g., bytes in UTF-8 encoding) of all N documents in the first intermediate corpus are concatenated sequentially according to document order to form a contiguous one-dimensional byte array (i.e., a continuous byte sequence). Simultaneously, an offset array of length N is recorded, where the i-th element represents the starting position (i.e., the starting offset) of the i-th document within the continuous byte sequence. After completing the concatenation and offset recording, the continuous byte sequence and the corresponding starting offsets for each document are transferred from CPU memory to GPU video memory for direct reading by the graphics processing unit (GPU) kernel during subsequent calls.

[0051] It should be noted that the above method of concatenating all documents into a continuous byte sequence and mapping it to video memory at once, compared with the traditional method of transferring each document to video memory in the form of discrete small data blocks, can significantly reduce the number of data transfers and transmission overhead. At the same time, it is beneficial for the graphics processor to efficiently read the word data of each document through continuous memory access in subsequent parallel computing.

[0052] Step a2: Start a two-dimensional thread grid on the graphics processor. Through a single kernel call to the two-dimensional thread grid, calculate the hash value of all the terms under each of the preset hash functions in parallel to obtain the calculation result of the hash value. The first dimension of the two-dimensional thread grid corresponds to the document number of each document in all the documents, and the second dimension of the two-dimensional thread grid corresponds to the term position of each term in the document. Each thread block in the two-dimensional thread grid contains multiple threads corresponding to the multiple preset hash functions.

[0053] In this embodiment of the application, after completing the data mapping shown in step a1, the electronic device can start a two-dimensional thread grid on the GPU, and uniformly schedule all threads in the two-dimensional thread grid through a single kernel call, so as to realize the fully parallel calculation of the hash values ​​of all words in all documents under all preset hash functions.

[0054] Specifically, the layout design of the aforementioned two-dimensional threaded grid is as follows: the first dimension of the grid (e.g., blockIdx.x) corresponds to the document number of each document in the first intermediate corpus, and the second dimension of the grid (e.g., blockIdx.y) corresponds to the position of each word within each document (i.e., which character triplet the word is in the document). Simultaneously, each thread block contains K threads, each corresponding to a preset hash function. Thus, the entire two-dimensional threaded grid starts a total of N×M_max×K threads (where N is the total number of documents, M_max is the maximum number of words in all documents, and K is the number of preset hash functions). Each thread's responsibility is to read a character triplet (i.e., a word consisting of 3 consecutive bytes) from the starting offset of the corresponding document based on its document number, calculate the hash value of the character triplet using its corresponding preset hash function, and finally write the calculation result into a three-dimensional memory tensor of shape [N, M_max, K].

[0055] In this embodiment, the aforementioned preset hash function can be a family of polynomial hash functions; wherein, for the i-th preset hash function, its functional form can be expressed as: h_i(x) = ((a_i ×x + b_i)mod p)mod 2^32, where the parameters a_i, b_i, and p are hash parameters that are randomly generated in advance and stored in the constant memory of the graphics processor. Since the constant memory is read-only for all threads and has extremely high access speed, each thread can directly read its corresponding hash parameter from the constant memory without repeatedly loading from the global video memory, thereby further reducing the memory access latency of hash calculation.

[0056] It should be noted that, since the actual number of terms varies across different documents, and the second dimension of the two-dimensional thread grid is uniformly set according to the maximum number of terms M_max across all documents, the hash values ​​generated by the thread positions (hereinafter referred to as padding positions) that exceed the actual number of terms in a document are invalid. The handling of these padding positions will be explained in conjunction with the minimum value reduction operation in subsequent steps b1-b2.

[0057] Here, by using the above-mentioned method of parallel calculation of the hash values ​​of all words in all documents under all preset hash functions in a single kernel call, this embodiment of the application transforms the triple nested loop calculation that the CPU needs to execute serially on a document-by-document, word-by-word, and hash function-by-hash-function basis in the prior art into a fully parallel single-step calculation on the graphics processor. This allows the calculation of hundreds of millions of hash values ​​in the entire corpus to be completed in a very small number of graphics processor clock cycles, and the calculation efficiency is improved by orders of magnitude compared with the pure CPU solution shown in the prior art.

[0058] Specifically, as an optional embodiment, the signature matrix reduction generation process in step S102 can be performed according to the method shown in steps b1-b2 below: Step b1: Write the calculation result of the hash value into a memory tensor of a first shape; wherein, the first shape is a multi-dimensional shape composed of the number of documents in all documents, the maximum number of tokens contained in a single document in all documents, and the number of preset hash functions.

[0059] In this embodiment, after parallel hash value calculation is completed in step a2, the calculated hash values ​​are uniformly written into a three-dimensional memory tensor of shape [N, M_max, K]. This three-dimensional memory tensor is the memory tensor of the first shape mentioned above. The first dimension N of this three-dimensional memory tensor corresponds to the total number of documents in the first intermediate corpus, the second dimension M_max corresponds to the maximum number of tokens contained in a single document, and the third dimension K corresponds to the number of multiple preset hash functions. Each element out_hash[doc_id, ngram_i, fn_idx] in this memory tensor represents the hash value of the nth token of the nth gram_i term in the doc_id document under the fn_idx preset hash function.

[0060] It should be noted that for the fill positions mentioned in step a2, the element values ​​at the corresponding positions in the aforementioned 3D memory tensor are all invalid values. In this embodiment, before performing the subsequent minimum reduction operation on the 3D memory tensor, the element values ​​corresponding to the fill positions can be replaced with a preset maximum value (for example, setting the element value at that position to the maximum value of the data type supported by the graphics processor), so that these fill positions will not be incorrectly selected as valid minimum values ​​during the subsequent minimum reduction process. Alternatively, a fill mask tensor with the same shape as the 3D memory tensor can be used to mark the validity of each position, and in the reduction operation, only the element values ​​at valid positions can be compared based on this mask.

[0061] Step b2: Perform minimum reduction on the memory tensor along the lexical dimension corresponding to the maximum number of lexical units to obtain a signature matrix of the second shape as the target signature matrix; wherein, the second shape is a multidimensional shape composed of the number of documents in all documents and the number of preset hash functions.

[0062] In this embodiment, after obtaining the memory tensor of the first shape (i.e., the three-dimensional memory tensor of shape [N, M_max, K]), the electronic device can perform a minimum value reduction operation along the second dimension of the memory tensor (i.e., the word dimension corresponding to the maximum number of words M_max) within the GPU's memory. The minimum value reduction operation means that for each preset hash function of each document, the smallest hash value calculated on all words (a total of M_max words, including valid words and padding words) of the document is selected as the final signature value of the document under that preset hash function.

[0063] Specifically, the above-mentioned minimum reduction operation can be performed by calling efficient parallel reduction primitives supported by the graphics processing unit (such as the torch.min function in the PyTorch framework, which uses CUDA's built-in warp-level shuffle instruction to achieve logarithmic time complexity tree reduction). During the reduction process, the graphics processing unit utilizes its massively parallel computing capabilities to simultaneously calculate local minima for each sub-region in the video memory tensor and reduces it upwards level by level until the global minimum is obtained.

[0064] After the aforementioned minimum reduction along the lexical dimension, the original three-dimensional memory tensor [N, M_max, K] is compressed into a two-dimensional matrix of shape [N, K]. This two-dimensional matrix is ​​the signature matrix of the second shape mentioned above, and in this embodiment, it is used as the target signature matrix corresponding to the first intermediate corpus. Each row in the target signature matrix represents the target hash signature of each document in all documents; that is, each document is represented by a signature vector of length K, and the j-th element in this signature vector is the result of taking the minimum value of the j-th preset hash function over all lexical terms of that document.

[0065] According to the mathematical properties of the MinHash algorithm, the proportion of documents whose target hash signatures have equal values ​​at corresponding positions is an unbiased estimate of the Jaccard similarity between the two documents. Therefore, using the aforementioned target signature matrix, local sensitive hashing can be efficiently performed in the subsequent step S103 to quickly identify duplicate documents in the first intermediate corpus whose similarity exceeds a threshold.

[0066] It should be noted that the entire processing of steps a1-a2 and b1-b2 described above is completed in a closed loop within the graphics processor's (GPU) memory. From reading consecutive byte sequences and parallel calculation of hash values ​​to minimizing the GPU tensor, all intermediate data remains within the GPU's memory, without any data transfer between the GPU and the CPU. Compared to existing technologies that copy the signature matrix from GPU memory back to CPU memory before subsequent bucketing operations, this completely eliminates the I / O bottleneck caused by CPU-GPU data transfer, and is one of the key technical means to significantly improve overall deduplication efficiency.

[0067] S103, in the graphics processor's video memory, the target signature matrix is ​​subjected to local sensitive hashing bucketing to determine the duplicate documents contained in the first intermediate corpus; and based on the duplicate documents, the first intermediate corpus is deduplicated to obtain the deduplicated first intermediate corpus as the second intermediate corpus.

[0068] Here, step S103 also belongs to the deduplication process in the post-stage of the weight operator mentioned above. Specifically, step S103 takes the target signature matrix generated in step S102 and performs local sensitive hashing and deduplication processing on the target signature matrix in the graphics processor's video memory.

[0069] Specifically, in step S102, the K-dimensional target hash signatures corresponding to all N documents in the first intermediate corpus have been obtained. If the duplicate documents are determined by directly comparing the signature vectors pairwise, the computational complexity is O(N²), which will result in unacceptable computational overhead when N reaches the millions. Therefore, in step S103, this embodiment of the application adopts the locality-sensitive hashing (LSH) bucketing technique. By dividing the signature vector into multiple strip-shaped sub-matrices and bucketing them separately, potential duplicate documents are grouped into the same bucket, thereby narrowing the search range of similar document pairs from all document pairs to document pairs within the same bucket, reducing the computational complexity from O(N²) to close to O(N).

[0070] It should be noted that, unlike the existing technology that uses data structures such as Python dictionaries for serial bucketing on the CPU, the embodiments of this application keep the entire process of locality-sensitive hashing bucketing—including strip submatrix partitioning, row hash calculation, sorting, and adjacent equality detection—executed in a closed loop within the graphics processor's video memory. This allows the bucketing operation to fully utilize the large-scale parallel computing capabilities of the graphics processor, while completely avoiding the data transfer overhead caused by transferring the signature matrix back from video memory to CPU memory.

[0071] Specifically, as an optional embodiment, the process of performing local sensitive hashing on the target signature matrix and identifying duplicate documents in step S103 can be performed according to the method shown in steps c1-c4 below: Step c1: Divide the target signature matrix into multiple strip sub-matrices along the column direction; wherein, the column direction is the dimension direction corresponding to the number of the preset hash functions in the target signature matrix, and the strip sub-matrices contain multiple consecutive columns of the target signature matrix.

[0072] In this embodiment of the application, the shape of the target signature matrix is ​​[N, K], where N is the total number of documents in the first intermediate corpus, and K is the number of preset hash functions (i.e., the dimension of the signature vector of each document); wherein, the column direction mentioned above refers to the dimension direction corresponding to the number K of preset hash functions in the target signature matrix.

[0073] Specifically, dividing the target signature matrix into multiple band sub-matrices along the aforementioned column direction means: dividing the K dimensions of the signature vector into b band sub-matrices (i.e., bands) along the column direction. Each band sub-matrix contains r consecutive columns from the target signature matrix (i.e., each band contains a consecutive segment of hash values ​​from the signature vector), and satisfies the relationship K = b × r. For example, when K = 128, it can be divided into b = 16 band sub-matrices, each containing r = 8 consecutive columns.

[0074] It should be noted that the number of sub-divisions b and the number of consecutive columns r in each sub-division can be flexibly configured according to the balance requirements of deduplication accuracy and deduplication efficiency in actual application scenarios. This application embodiment does not impose specific limitations on this.

[0075] Step c2: For each of the strip sub-matrices, calculate the row hash value for each row in the strip sub-matrices to obtain the strip hash value vector corresponding to the strip sub-matrices.

[0076] In this embodiment of the application, for each strip submatrix (of shape [N, r]) obtained in step c1, a "row hash value calculation" is performed on each row of the strip submatrix (i.e., the r-dimensional signature fragment of each document in the strip submatrix).

[0077] Specifically, the above row hash value calculation refers to taking the r hash values ​​of the row as input and mapping the r-dimensional vector into a scalar hash value (i.e., bucket number) through a preset row hash function. At this time, for each strip submatrix, a one-dimensional vector of length N can be obtained. This vector is the strip hash value vector corresponding to the strip submatrix, and each element in the strip hash value vector represents the bucket number of the corresponding document in the strip submatrix.

[0078] It should be noted that the row hash function described above compresses the r-dimensional signature fragments of all documents within a band into a single scalar value, ensuring that documents with similar signature fragments within that band have a high probability of being mapped to the same bucket number. According to the theoretical properties of locality-sensitive hashing, the probability that two documents within the same band have completely identical signature fragments is equal to the r-th power of their Jaccard (Jaccard Similarity Coefficient) similarity. Therefore, by appropriately setting the value of r, the balance between recall and precision for duplicate documents can be effectively controlled.

[0079] Step c3: Sort each of the striped hash value vectors according to the ascending or descending order of the hash value to obtain the sorting result.

[0080] In this embodiment of the application, after obtaining the strip hash value vector corresponding to each strip submatrix, the hash values ​​in the strip hash value vector can be sorted in ascending or descending order in the graphics processor's video memory. The purpose of sorting is to gather documents with the same hash value (i.e., falling into the same bucket) into adjacent positions in the sorting result, thereby providing a basis for quickly finding duplicate documents by adjacent comparison in the subsequent step c4.

[0081] It should be noted that within the GPU graphics processor, efficient parallel sorting algorithms (such as the argsort sorting algorithm) can also be called to complete the sorting operation in step c3 above. In this case, the sorting efficiency is much higher than that of the serial dictionary insertion operation in the CPU.

[0082] Step c4: From the sorting results, determine the documents corresponding to the hash values ​​that are adjacent in position and equal in value as the duplicate documents.

[0083] In this embodiment of the application, after sorting each strip hash value vector, the electronic device can compare the hash values ​​of adjacent positions in the sorting result: if the hash values ​​of two adjacent positions are equal, it indicates that the two documents corresponding to the two adjacent hash values ​​fall into the same bucket in the strip sub-matrix (that is, the signature fragments of the two documents on the strip sub-matrix are completely consistent). At this time, according to the theoretical properties of the MinHash algorithm, it can be determined that the two documents have a high Jaccard similarity and should be identified as duplicate documents.

[0084] It should be noted that since documents with equal hash values ​​will be arranged consecutively in the sorted results, all document groups falling into the same bucket in all strip sub-matrices can be found by performing a single traversal of the sorted results and comparing the hash values ​​of adjacent elements one by one. Performing the above operation for each strip sub-matrix and summing up the duplicate documents found in each strip sub-matrix will yield all the duplicate documents contained in the first intermediate corpus.

[0085] Specifically, after identifying the duplicate documents in the first intermediate corpus, deduplication can be performed on the first intermediate corpus based on the identified duplicate documents. The specific deduplication strategy used can be as follows: Based on the identified duplicate documents, a connectivity graph is constructed. Documents that are interconnected in the first intermediate corpus are grouped into the same duplicate document group. Then, one document is selected from each duplicate document group as the retained document (e.g., the document with the longest text length, lowest perplexity, or earliest appearance). The remaining documents are marked for deletion. Finally, based on the above marking results, the retained documents are extracted from the first intermediate corpus, resulting in the deduplicated first intermediate corpus, which serves as the second intermediate corpus.

[0086] It should be noted that the entire process of S103—including strip submatrix partitioning, row hash value calculation, sorting, adjacent equality detection, and deduplication—is completed in a closed loop within the graphics processor's (GPU) memory. There is no need to transfer the signature matrix or intermediate hash value vector from GPU memory back to CPU memory. Compared to existing technologies that transfer the signature matrix back to the CPU and then use the CPU-side dictionary data structure for serial bucketing, this method not only significantly improves bucketing efficiency through the GPU's parallel sorting capabilities but also fundamentally eliminates the I / O bottleneck caused by frequent data transfers between GPU memory and CPU memory. This is one of the key technical means to achieve efficient deduplication of massive corpora.

[0087] S104, the second intermediate corpus is input into a pre-trained document classification model, and the document classification model is used to predict the document category of each document in the second intermediate corpus, thereby obtaining the original classification logical value corresponding to each document in the second intermediate corpus.

[0088] Here, step S104 belongs to the document classification part in the post-stage of the weight operator mentioned above; after obtaining the deduplicated second intermediate corpus (that is, the second intermediate corpus that does not contain duplicate documents), considering that the large language model needs to match the professional knowledge under different industry fields when processing documents in different industry fields, in this embodiment of the application, it is also necessary to perform classification prediction on the documents in the second intermediate corpus by the document classification method shown in step S104, so as to obtain the original classification logic value corresponding to each document and each preset document category.

[0089] It should be noted that, unlike the document classification methods commonly used in the prior art (which employ extremely simple statistical methods, such as TF-IDF combined with Naive Bayes; or directly call a general large model for zero-shot classification), the embodiments of this application use a dedicated document classification model obtained by supervised fine-tuning based on pre-trained language models such as bert-base-chinese, which can ensure high classification accuracy while taking into account the processing efficiency of TB-level corpora.

[0090] The pre-training process of the above document classification model will be explained in detail first, and then the process of using the document classification model to perform inference and obtain the original classification logic value in step S104 will be explained.

[0091] The pre-training process for document classification models: In this embodiment, the pre-trained document classification model is obtained by supervised fine-tuning of the base model. The base model can adopt the BERT-base-chinese model structure, which contains a 12-layer Transformer structure with a hidden dimension of 768 and a total of approximately 110 million parameters.

[0092] Specifically, the core computational mechanism of the aforementioned basic model is a multi-head self-attention mechanism: Attention(Q, K,V) = softmax(Q×K^T / )×V, where Q (query vector), K (key vector), and V (value vector) are obtained from the model input X through linear transformation, and d_k is the dimension of the key vector K, divided by The purpose is to prevent the gradient from vanishing due to excessively large dot product values.

[0093] Here, regarding the construction of model training data, as an optional embodiment, those skilled in the art can manually filter and extract multi-domain professional texts from open-source general corpora (such as CLUE, etc.) by domain. The industry domains covered can include at least one of the professional fields such as law, medicine, and automobiles, as well as a general category. By labeling each obtained text with its corresponding real document category label, the model training data can be constructed. For each text in the model training dataset, the bert-base-chinese WordPiece word segmenter can be used for word segmentation, dividing the text into sub-word units, and adding special markers [CLS] and [SEP] to the beginning and end of the sequence, respectively. After truncating or padding to a fixed length (e.g., max_length=512), a training sample in triplet format containing input_ids, attention_mask, and labels can be created.

[0094] Specifically, during the model fine-tuning training phase, based on the aforementioned basic model, a linear classification layer can be added after the output hidden state vector h_CLS at the [CLS] position: logits = W_c × h_CLS + b_c, where W_c is a weight matrix of shape [C × 768] (C is the total number of preset document categories), and b_c is the bias vector. The model training uses the cross-entropy loss function to minimize the difference between the model's predicted probability distribution and the true document category labels. During actual model training, fine-tuning frameworks such as MS-Swift can be used, setting the learning rate to 2e-5, and employing a warmup and linear decay learning rate scheduling strategy. Training is performed for 3 to 5 epochs, and the optimal model checkpoint is selected on the validation set using the F1 score as the metric, becoming the final pre-trained document classification model.

[0095] It should be noted that the above-mentioned basic model can also be replaced by models such as RoBERTa-wwm-ext or MacBERT, which have more in-depth Chinese pre-training; the fine-tuning strategy can also adopt low-rank adaptation methods such as LoRA, so as to maintain a near-full parameter fine-tuning effect while significantly reducing memory usage. This application does not specifically limit the implementation of this method.

[0096] Regarding the process in step S104 of using the above document classification model to perform reasoning and obtain the original classification logical value: After the pre-training of the document classification model is completed, in step S104, each document in the second intermediate corpus obtained in step S103 is input into the pre-trained document classification model. After multi-layer Transformer forward inference of the model, the original classification logical value corresponding to each document is finally obtained in the output layer. The original classification logical value is a vector composed of the unnormalized scores output by the document classification model corresponding to each preset document category.

[0097] Specifically, for a document in an input document classification model, the document classification model outputs a vector of length C (where C is the total number of preset document categories). Each element in this vector corresponds to the original score value of a preset document category (e.g., automotive, medical, legal, general, etc.). The higher the original score value of a preset document category, the greater the likelihood that the model will classify the document into that preset document category. However, since this vector has not undergone softmax normalization, its value cannot be directly interpreted as a probability value. Therefore, in this embodiment of the application, it is defined as the "original classification logical value".

[0098] It should be noted that the original classification logic values ​​will be further calibrated in the GPU's video memory in subsequent step S105 to obtain the calibrated confidence score for each document, thereby achieving a quantitative assessment of the reliability of the document classification results. This embodiment retains the original classification logic values ​​generated by inference in the GPU's video memory instead of sending them back to the CPU. This is a key prerequisite for achieving efficient confidence filtering, avoiding the I / O overhead caused by frequent transfer of the logits tensor between the GPU and CPU in existing technologies, thus significantly improving processing throughput.

[0099] S105, in the video memory of the graphics processor, the confidence level of the original classification logic value is calibrated to obtain the confidence level calibration result corresponding to each document in the second intermediate corpus, and the confidence level category to which each document in the second intermediate corpus belongs is determined according to the confidence level calibration result and the preset multi-level confidence level threshold.

[0100] Here, step S105 belongs to the confidence filtering part of the post-stage of the aforementioned weight operator; in step S104, the original classification logical values ​​corresponding to each document in the second intermediate corpus have been obtained through the pre-trained document classification model. However, the original classification logical values ​​are only unnormalized raw score vectors, which cannot be directly interpreted as the probability of a document belonging to each category, nor have they undergone any reliability verification.

[0101] In existing technologies, the category with the highest score in the original classification logic values ​​is usually directly taken as the final classification result. This lack of quantitative evaluation of the confidence level of the classification result leads to a large number of boundary samples being misclassified due to model misjudgment, affecting the purity and quality of the final corpus. To address this problem, in step S105 of this application embodiment, confidence calibration and multi-level confidence threshold routing filtering are sequentially performed on the original classification logic values ​​within the graphics processor's memory to quantitatively evaluate the reliability of each document's classification result and determine the confidence category to which each document belongs.

[0102] It is important to note that, unlike the existing technology where the logits tensor generated by inference is transferred back from the graphics processor's memory to the CPU memory, and then softmax normalization, confidence calculation, and threshold determination are performed sequentially on the CPU, this embodiment maintains the entire process of confidence calibration and threshold routing within the graphics processor's memory in a closed loop. Only the final routing decision result is returned to the CPU in the form of a lightweight index. This eliminates redundant data transfer overhead between the CPU and GPU while fully utilizing the parallel computing capabilities of the graphics processor to accelerate the post-processing. The specific implementation process of step S105 is described in detail below.

[0103] Specifically, step S105 can be further divided into the following sub-steps: the confidence calibration process shown in steps d1-d2, the multi-level confidence threshold routing filtering process shown in steps e1-e2, and the lightweight index generation and transmission process shown in steps f1-f2. The specific implementation methods of each of the above sub-steps are described in detail below: Regarding the confidence calibration process: In this embodiment of the application, the purpose of the confidence calibration process is to post-process the original classification logic value output in step S104, convert it into a confidence value with probabilistic significance, and in this process, correct the "overconfidence" problem that may exist in the document classification model through the temperature scaling mechanism shown in steps d1-d2 below.

[0104] Step d1: For each document in the second intermediate corpus, divide the original classification logic value corresponding to the document by a preset temperature parameter to obtain the temperature scaling calibration result of the original classification logic value.

[0105] In this embodiment, for each document in the second intermediate corpus, the original classification logic value obtained in step S104 is a vector of length C (where C is the total number of preset document categories). Each element in this vector represents the unnormalized original score of the document belonging to the corresponding preset document category. Because neural network models may be "overconfident" in their predictions during training (i.e., the model may output extremely high confidence scores even for some incorrectly predicted samples), the confidence score obtained by directly performing softmax normalization on the original classification logic value often fails to accurately reflect the actual probability of the model's correct prediction. Therefore, this embodiment introduces a temperature scaling mechanism, dividing the original classification logic value by a preset temperature parameter T to obtain a temperature scaling calibration result.

[0106] The mathematical principle behind the temperature scaling is as follows: Let the original classification logic value corresponding to a document be a logits vector, and the preset temperature parameter be T. Then, the temperature-scaled result is logits / T. When the temperature parameter T > 1, the relative differences between elements in the original classification logic value are reduced, and the probability distribution after softmax normalization becomes smoother. The original high confidence scores are moderately suppressed, thereby alleviating the problem of the model being "overconfident" in its own predictions. When T = 1, it degenerates into the original softmax output; when T < 1, the probability distribution becomes sharper. In the embodiments of this application, the temperature parameter T is usually taken to be greater than 1 so that the calibrated confidence score can more realistically reflect the actual probability of the model's correct prediction.

[0107] It should be noted that the above-mentioned preset temperature parameters are scalar values ​​that were pre-determined by optimizing on the model validation dataset before the formal corpus cleaning process started. Once determined, they remain fixed in the subsequent cleaning process and do not need to be recalculated for each inference.

[0108] Specifically, before performing step d1, the preset temperature parameter can be determined in advance using the methods shown in steps 1-5: Step 1: Obtain multiple sample documents and the true document category to which each sample document belongs from the model validation dataset of the document classification model.

[0109] In this embodiment, the model validation dataset is a portion of labeled data that was not used for model training and was separated from the original training data. Each sample document in the model validation dataset has a known real document category label (e.g., automotive, medical, legal, general, etc.), therefore, it can be used as a standard to evaluate the effectiveness of subsequent temperature parameter calibration.

[0110] Step 2: For each sample document, input the sample document into the document classification model, predict the document category to which the sample document belongs through the document classification model, and obtain the original classification logical value corresponding to the sample document.

[0111] In this embodiment, the operation of this step is the same as the reasoning method for the second intermediate corpus in step S104. After each sample document undergoes forward reasoning through multiple Transformers of the document classification model, an unnormalized score vector of length C is obtained at the output layer. This vector is the original classification logical value corresponding to the sample document.

[0112] Step 3: For each candidate temperature parameter among multiple candidate temperature parameters, divide the original classification logical value corresponding to the sample document by the candidate temperature parameter to obtain the target temperature scaling calibration result. Then, normalize the target temperature scaling calibration result using the softmax function to obtain the calibration prediction probability of the sample document for the real document category under the candidate temperature parameter.

[0113] In this embodiment, the aforementioned candidate temperature parameters can be obtained by sampling within a preset temperature range (e.g., [1.0, 10.0]) at predetermined steps (e.g., 0.5), thereby obtaining a set of candidate temperature parameters to be evaluated. For each candidate temperature parameter T_candidate, following the same operation as steps d1 and d2, the original classification logical value of the sample document is first divided by T_candidate, and then the division result is normalized using the softmax function. The probability value corresponding to the true document category of the sample document is extracted from the normalized probability vector. This probability value is the calibration prediction probability of the sample document for the true document category under the candidate temperature parameter, reflecting the confidence level of the model in correctly classifying the sample document under the temperature parameter.

[0114] Step 4: Calculate the loss function value corresponding to the candidate temperature parameter based on the calibration prediction probability corresponding to each sample document under the candidate temperature parameter.

[0115] In this embodiment, the loss function can be a negative log-likelihood loss function. Specifically, the calculation is as follows: Take the natural logarithm of the calibration prediction probability of each sample document for the true document category under the candidate temperature parameter, then take the negative of the logarithmic value to obtain the negative log-likelihood value for each sample document; then, average the negative log-likelihood values ​​of all sample documents. This average value is the negative log-likelihood loss function value corresponding to the candidate temperature parameter. A smaller loss function value indicates that, under the candidate temperature parameter, the model's overall prediction probability for the true category of each sample in the entire validation set is higher, meaning the temperature calibration effect is better.

[0116] Step 5: From the multiple candidate temperature parameters, determine the candidate temperature parameter with the smallest loss function value as the preset temperature parameter.

[0117] In this embodiment, after calculating the loss function values ​​for all candidate temperature parameters, the magnitudes of the loss function values ​​corresponding to each candidate temperature parameter are compared, and the candidate temperature parameter with the smallest loss function value is selected as the final preset temperature parameter T. This preset temperature parameter T is an optimized and verified scalar value that can maximize the improvement of the confidence calibration effect of the original classification logic value, making the calibrated confidence score more realistic and reliable.

[0118] It should be noted that the above method of traversing multiple candidate temperature parameters and selecting the candidate temperature parameter with the smallest loss function value as the preset temperature parameter is only a preferred embodiment provided in this application. In alternative embodiments, other confidence calibration methods such as isofrequency histogram calibration or Platt calibration can also be used, as long as the reliability of the original classification logic value can be corrected and the calibrated confidence level can be output. This application does not specifically limit this method.

[0119] Step d2: Normalize the temperature scaling calibration result using the softmax function, and use the normalized result as the confidence calibration result corresponding to the document.

[0120] In this embodiment of the application, after completing the temperature scaling in step d1, the softmax function is called to normalize the temperature scaling calibration result vector of each document.

[0121] Specifically, the softmax function is calculated as follows: P(y=k|X) = softmax(logits_k / T) / Σ_jexp(logits_j / T), where logits_k / T represents the score of the document corresponding to the k-th preset document category after temperature scaling, and Σ_jexp(logits_j / T) represents the sum of the exponents of the scores of all preset document categories after temperature scaling.

[0122] It should be noted that after normalization using the softmax function, the sum of all elements in the output vector is 1. Each element value can be interpreted as the calibrated probability value of the document belonging to the corresponding preset document category. This normalized vector is the confidence calibration result for that document.

[0123] It should be noted that the temperature scaling and softmax normalization operations in steps d1 and d2 are performed directly within the graphics processor's video memory. Specifically, after obtaining the original classification logic values ​​from each document in a batch, these values ​​are stored in the graphics processor's video memory as tensors. Then, scalar division (divided by a preset temperature parameter T) and softmax normalization are performed on this tensor within the video memory. Throughout this process, the original classification logic values ​​and the intermediate data of the calibration process do not leave the graphics processor's video memory. Compared to the existing technology that sends the logits tensor back to the CPU and then calculates the softmax value line by line on the CPU, this avoids the I / O overhead caused by data being moved back and forth between the graphics processor and the CPU, significantly improving processing efficiency.

[0124] Regarding the multi-level confidence threshold routing filtering process: The purpose of the multi-level confidence threshold routing filtering process is to classify each document into different confidence categories based on the comparison between the confidence calibration results and the preset multi-level confidence thresholds, so that subsequent steps S106 can adopt differentiated processing strategies according to different confidence categories.

[0125] Here, the multi-level confidence threshold includes at least a low confidence threshold θ_low; in addition, the multi-level confidence threshold may also include a high confidence threshold θ_high (where the low confidence threshold θ_low < the high confidence threshold θ_high); wherein, the multi-level confidence threshold routing filtering process is as follows: steps e1-e2: Step e1: For each document in the second intermediate corpus, determine whether the confidence calibration result corresponding to the document is less than or equal to the low confidence threshold.

[0126] In this embodiment, the confidence calibration result corresponding to the document refers to the maximum value in the calibrated confidence vector (i.e., the confidence calibration result of the original classification logic value) output by the aforementioned confidence calibration step, which is the highest calibrated confidence score of the model for the category to which the document belongs (hereinafter referred to as the document's confidence score). For example, if the probability vector (i.e., the confidence calibration result of the original classification logic value) of a document after confidence calibration is [0.55, 0.10, 0.22, 0.13], then the document's confidence score is 0.55.

[0127] Specifically, in step e1, the confidence score can be compared with a preset low confidence threshold θ_low (e.g., 0.60) to determine whether it is less than or equal to the low confidence threshold.

[0128] Step e2: If the confidence calibration result corresponding to the document is less than or equal to the low confidence threshold, then the document is determined to belong to the low confidence category; wherein, the low confidence category is a confidence category that is lower than the preset standard.

[0129] In this embodiment of the application, if the confidence score of the document is less than or equal to the low confidence threshold θ_low, it indicates that the model has very low confidence in the classification prediction result of the document, the classification result of the document is unreliable, and it should be classified into the low confidence category.

[0130] Furthermore, if the confidence score of the document is greater than the low confidence threshold θ_low and less than the high confidence threshold θ_high (e.g., 0.90), it indicates that the model has some confidence in the classification prediction of the document, but not enough confidence. It should be classified into the medium confidence category and further confirmed through manual review or other means.

[0131] If the confidence score of the document is greater than or equal to the high confidence threshold θ_high, it indicates that the model has a high degree of confidence in the classification prediction of the document, and it should be classified into the high confidence category and can be directly used as a high-quality target corpus.

[0132] It should be noted that the process of determining the confidence category based on the comparison between the confidence calibration results and the multi-level confidence thresholds, as described above, is also executed within the GPU's video memory. Specifically, after obtaining the confidence calibration results for each document, the threshold comparison and category determination are performed simultaneously on all documents in the entire batch in parallel using GPU tensor operation primitives (such as element-wise comparison operations and Boolean mask operations) within the GPU's video memory. This eliminates the need for processing each document individually through Python loops, further improving processing efficiency.

[0133] Regarding the lightweight index generation and transmission process: The purpose of the lightweight index generation and transmission process is to send the classification decision results (including predicted category and confidence category) of each document from the graphics processor back to the CPU with a very small amount of data, so that the CPU can perform the filtering operation in the subsequent step S106 based on the lightweight index, while avoiding sending a large amount of intermediate calculation data in the graphics processor's video memory back to the CPU.

[0134] Specifically, after determining the confidence category to which each document in the second intermediate corpus belongs, as an optional embodiment, the electronic device can perform the above-described lightweight index generation and transmission process according to the method shown in steps f1-f2 below: Step f1: Determine the category prediction result for each document in the second intermediate corpus from the original classification logic values ​​corresponding to each document in the second intermediate corpus; wherein the category prediction result is the preset document category corresponding to the largest prediction probability value among the original classification logic values.

[0135] In this embodiment, for each document in the second intermediate corpus, after confidence calibration, the preset document category (e.g., automotive, medical, legal, general, etc.) corresponding to the dimension with the highest probability value can be taken from the softmax-normalized probability vector (i.e., the confidence calibration result of the original classification logical values) as the category prediction result for that document. Alternatively, the preset document category corresponding to the dimension with the highest score can be directly taken from the original classification logical values ​​as the category prediction result. The results of both are consistent under the argmax operation (because softmax normalization does not change the relative order of the elements).

[0136] Step f2: Use the category prediction results and the confidence category corresponding to each document in the second intermediate corpus as a lightweight index corresponding to each document in the second intermediate corpus, and transmit the lightweight index to the central processing unit.

[0137] In this embodiment, the lightweight index is specifically structured as follows: for each document, two integer values ​​(or enumerated values) represent which preset document category the document is predicted to belong to (i.e., the category prediction result) and which confidence category the document is assigned to (i.e., high, medium, or low confidence category). Since each document only needs to transmit these two lightweight decision conclusions, compared to the prior art method of transmitting the entire original classification logic value (containing C floating-point numbers, each occupying 4 bytes) from the graphics processor back to the CPU, the amount of data transmission is significantly reduced.

[0138] For example, taking a preset number of document categories C=10 and a batch size of 256 as an example, the prior art requires the transmission of approximately 256×10×4=10KB of logits floating-point data per batch, while the embodiment of this application only requires the transmission of approximately 256×2×4=2KB (or less, if enumerated value compression is used) of lightweight index data, reducing the data transmission volume by approximately 80% to 90%, which greatly alleviates the data transmission bandwidth pressure between the graphics processor and the CPU.

[0139] It should be noted that the confidence calibration, multi-level threshold routing filtering, and lightweight index generation processes in step S105 are all executed in a closed loop within the graphics processor's video memory. In step f2, only the final lightweight index—that is, the category prediction results and confidence category identification information for each document—is transferred from the graphics processor to the CPU. Intermediate calculation data such as the original classification logic values, temperature scaling calibration results, and the complete probability vector after softmax normalization are retained in the graphics processor's video memory and not transmitted back to the CPU. This allows the embodiments of this application to achieve confidence assessment and routing filtering functions while minimizing the data transfer overhead between the graphics processor and the CPU, making it one of the key technical means for achieving efficient confidence filtering of massive corpora.

[0140] S106, From the second intermediate corpus, documents with confidence levels below a preset standard are filtered out to obtain the remaining documents as target corpus for training the large language model.

[0141] In this embodiment, step S106 is the final screening step in the confidence filtering part of the post-stage of the weight operator. In step S105, confidence category (e.g., high confidence, medium confidence, or low confidence) has been determined for each document in the second intermediate corpus through confidence calibration and multi-level confidence threshold routing filtering. Step S106, based on step S105, performs differentiated screening according to the confidence category of each document to obtain high-quality target corpus that can be used for training large language models.

[0142] Specifically, the aforementioned preset standard is a pre-defined minimum confidence requirement used to distinguish whether a document can be directly adopted as training corpus. In this embodiment, the preset standard may correspond to the low confidence threshold among the aforementioned multi-level confidence thresholds: documents classified into the low confidence category (i.e., documents with confidence scores less than or equal to the low confidence threshold) are considered unreliable due to the model's extremely low confidence in their classification predictions and should be discarded.

[0143] Furthermore, documents classified into the high-confidence category (i.e., documents with a confidence score greater than or equal to the high-confidence threshold) can be directly retained as target corpora for the corresponding industry's professional domain, since the model has a high degree of confidence in its classification predictions. Documents classified into the medium-confidence category (i.e., documents with a confidence score between the low-confidence and high-confidence thresholds), while having some reference value, are not entirely reliable and can be sent to a manual review queue for further review and confirmation by human annotators before a decision is made on whether to retain or discard them.

[0144] It should be noted that the screening process shown in step S106 can be executed within the CPU. After receiving the lightweight index (i.e., category prediction result and confidence category) corresponding to each document in the second intermediate corpus in step f2, the CPU can extract the documents that meet the retention conditions (i.e., documents with high confidence categories) from the second intermediate corpus based on the comparison results of the confidence category of each document with the preset standard, and discard the documents that do not meet the retention conditions (i.e., documents with low confidence categories), and finally obtain the remaining documents as the target corpus for training the large language model.

[0145] Through the document classification and confidence filtering process shown in steps S104 to S106 above, this embodiment of the application not only realizes the industry category division of each document in the second intermediate corpus, but also performs quantitative evaluation and graded processing of the reliability of each classification result through confidence calibration and multi-level threshold routing filtering mechanism. In this way, while ensuring classification accuracy, it effectively selects high-confidence, high-quality corpus as the final training data, significantly improving the purity and quality of the target corpus used to train the large language model.

[0146] In a preferred embodiment of this application, in order to further improve the overall processing efficiency of corpus cleaning, the document classification reasoning, confidence calibration and subsequent screening operations involved in the above steps S104 to S106 can be executed in a multi-CUDA pipeline mode to eliminate the idle waiting time of the graphics processor in the data preparation stage.

[0147] In this embodiment, the corpus cleaning method further includes a multi-CUDA pipeline execution step. Specifically, through a first CUDA stream, the document category prediction and confidence calibration operations of the document classification model are performed on the current batch of the second intermediate corpus; simultaneously, the central processing unit preprocesses the next batch of the second intermediate corpus to obtain target second intermediate corpus that conforms to the input format of the document classification prediction model, and through a second CUDA stream, the target second intermediate corpus is asynchronously transmitted to the document classification model as the model input data for the next batch; wherein, the document classification model is located in the video memory of the graphics processor.

[0148] It should be noted that in a graphics processing unit (GPU), a CUDA stream represents a sequence of GPU operations executed sequentially, and operations between different CUDA streams can be executed concurrently. In the embodiments of this application, at least two CUDA streams (e.g., a first CUDA stream and a second CUDA stream) are created, and the two CUDA streams process consecutive batches of data in an alternating manner.

[0149] Specifically, while the first CUDA stream is performing inference computation and confidence calibration for the document classification model on the second intermediate corpus of the current batch (e.g., batch i), the GPU's computing unit is busy. Simultaneously, the CPU does not idle but synchronously preprocesses the second intermediate corpus of the next batch (e.g., batch i+1). This preprocessing includes, but is not limited to: segmenting each document text using a tokenizer corresponding to the document classification model (e.g., the WordPiece tokenizer), converting the segmentation results into an input tensor format acceptable to the model (including input_ids and attention_mask), and truncating or padding the sequence to a fixed length. After completing the above preprocessing, the preprocessed target second intermediate corpus (i.e., the formatted model input tensor) is asynchronously transferred from CPU memory to the GPU's video memory via the second CUDA stream. Since the second CUDA stream and the first CUDA stream are independent, the above data transfer operations can be executed concurrently with the inference computation operations in the first CUDA stream without blocking each other.

[0150] Once the first CUDA stream completes its inference and confidence calibration for the current batch, the next batch of data in the second CUDA stream is ready and transferred to video memory. The graphics processor (GPU) can immediately switch to the second CUDA stream to begin inference for the next batch without waiting for the CPU to perform data preprocessing and data transfer. Similarly, while the second CUDA stream is performing inference, the first CUDA stream begins receiving preprocessed data for the next batch (e.g., the i+2th batch). This alternation ensures that the GPU's inference computation for the current batch overlaps with the CPU's data preprocessing and data transfer for the next batch, minimizing the GPU's idle time caused by waiting for data preparation.

[0151] It should be noted that, through the above-mentioned multi-CUDA pipeline execution method, with similar CPU preprocessing time and GPU inference time, the utilization rate of the graphics processor can be increased from about 50% under the traditional serial execution method to nearly 100%, and the overall batch processing throughput is almost doubled, further improving the processing efficiency in the scenario of cleaning massive corpora.

[0152] It should be noted that the above-described multi-CUDA pipeline execution method is a preferred implementation method provided by the embodiments of this application. It can be used in conjunction with the confidence calibration and lightweight index return mechanism described in step S105 above, or it can be applied independently to other scenarios that require batch inference. The embodiments of this application do not specifically limit this.

[0153] Based on the above-mentioned corpus cleaning method provided in the embodiments of this application, on the one hand, it can rationally utilize the parallel processing capability of the graphics processor, and migrate the deduplication operation in the corpus cleaning process to the graphics processor for parallel execution, effectively improving the deduplication efficiency; on the other hand, by performing confidence calibration and confidence filtering on the document classification results output by the document classification model, it can effectively remove documents with confidence categories lower than the preset standard during the data cleaning process, which is conducive to improving the corpus quality of the target corpus obtained after cleaning, thereby enabling this application to balance the efficiency and quality of corpus cleaning.

[0154] Based on the same inventive concept, this application also provides a corpus cleaning device corresponding to the above-mentioned corpus cleaning method. Since the principle of the corpus cleaning device in the embodiments of this application is similar to that of the above-mentioned corpus cleaning method in the embodiments of this application, the implementation of the corpus cleaning device can refer to the implementation of the above-mentioned corpus cleaning method, and the repeated parts will not be described again.

[0155] Reference Figure 2 As shown, Figure 2 This illustration shows a schematic diagram of a corpus cleaning apparatus provided in an embodiment of this application, wherein the corpus cleaning apparatus includes: The coarse screening module 201 is used to filter out texts that do not meet the preset quality conditions from the original corpus by performing a preset pre-filtering operation included in the coarse screening stage, and obtain a first intermediate corpus that meets the preset quality conditions. The deduplication module 202 is used to map all documents in the first intermediate corpus to the video memory of the graphics processor, and to calculate the hash values ​​of all words in all documents under multiple preset hash functions in a single kernel call of the graphics processor. Then, in the video memory of the graphics processor, the calculation results of the hash values ​​are reduced to the minimum value along the word dimension to obtain the target signature matrix corresponding to the first intermediate corpus. Each row in the target signature matrix represents the target hash signature of each document in the first intermediate corpus. In the graphics processor's video memory, the target signature matrix is ​​subjected to local sensitive hashing bucketing to determine the duplicate documents contained in the first intermediate corpus; and based on the duplicate documents, the first intermediate corpus is deduplicated to obtain the deduplicated first intermediate corpus as the second intermediate corpus. The classification module 203 is used to input the second intermediate corpus into a pre-trained document classification model, predict the document category of each document in the second intermediate corpus through the document classification model, and obtain the original classification logical value corresponding to each document in the second intermediate corpus; wherein, the original classification logical value is a vector composed of the unnormalized score output by the document classification model and corresponding to each preset document category; The confidence filtering module 204 is used to perform confidence calibration on the original classification logic value in the graphics processor's video memory to obtain the confidence calibration result corresponding to each document in the second intermediate corpus, and determine the confidence category to which each document in the second intermediate corpus belongs based on the confidence calibration result and the preset multi-level confidence threshold. From the second intermediate corpus, documents with confidence levels below a preset standard are filtered out, and the remaining documents are used as target corpus for training a large language model.

[0156] In one optional implementation, the preset pre-filtering operation includes at least one of the following: Filter out single text entries whose character count falls outside the preset character count range; Filter out individual texts whose language recognition results do not meet preset recognition conditions; wherein, the language recognition result is the probability value of the individual text belonging to the target language, which is input into the language recognition model and output by the language recognition model. Text standardization processing operations that convert a single text line into the target text format; Filter out individual texts whose text statistical features do not meet preset feature conditions; wherein, the text statistical features are determined based on at least one preset text statistical feature index; Filter out individual texts whose perplexity calculation result is higher than the preset perplexity threshold.

[0157] In an optional implementation, when mapping all documents in the first intermediate corpus to the graphics processor's video memory, and calculating the hash values ​​of all terms in all documents under multiple preset hash functions in parallel through a single kernel call of the graphics processor, the deduplication module 202 is used to: The bytes of all the documents are concatenated into a continuous byte sequence, and the continuous byte sequence and the starting offset of each document in the continuous byte sequence are mapped to the video memory of the graphics processor. A two-dimensional thread grid is launched on the graphics processor. The two-dimensional thread grid is called by a single kernel to calculate the hash value of all the terms under each preset hash function in parallel, and the calculation result of the hash value is obtained. The first dimension of the two-dimensional thread grid corresponds to the document number of each document in the total number of documents, and the second dimension of the two-dimensional thread grid corresponds to the term position of each term in the document. Each thread block in the two-dimensional thread grid contains multiple threads corresponding to the multiple preset hash functions.

[0158] In an optional implementation, when the hash value calculation result is reduced to a minimum value along the lexical dimension to obtain the target signature matrix corresponding to the first intermediate corpus, the deduplication module 202 is used to: The calculation result of the hash value is written into a memory tensor of a first shape; wherein, the first shape is a multi-dimensional shape composed of the number of documents in all documents, the maximum number of words contained in a single document in all documents, and the number of preset hash functions; The memory tensor is reduced to a minimum along the lexical dimension corresponding to the maximum number of lexical units to obtain a signature matrix of a second shape as the target signature matrix; wherein, the second shape is a multidimensional shape composed of the number of documents in all documents and the number of preset hash functions.

[0159] In an optional implementation, when performing local sensitive hashing on the target signature matrix to determine duplicate documents contained in the first intermediate corpus, the deduplication module 202 is used to: The target signature matrix is ​​divided into multiple strip sub-matrices along the column direction; wherein, the column direction is the dimension direction corresponding to the number of the preset hash functions in the target signature matrix, and the strip sub-matrices contain multiple consecutive columns of the target signature matrix; For each of the strip submatrixes, the row hash value is calculated for each row of the strip submatrix to obtain the strip hash value vector corresponding to the strip submatrix. Sort each of the strip hash value vectors according to the ascending or descending order of hash values ​​to obtain the sorting result; From the sorting results, documents with adjacent and equal hash values ​​are identified as the duplicate documents.

[0160] In an optional implementation, when performing confidence calibration on the original classification logic values ​​to obtain the confidence calibration results for each document in the second intermediate corpus, the confidence filtering module 204 is used to: For each document in the second intermediate corpus, the original classification logic value corresponding to the document is divided by a preset temperature parameter to obtain the temperature scaling calibration result of the original classification logic value. The temperature scaling calibration result is normalized using the softmax function, and the normalized result is used as the confidence calibration result corresponding to the document.

[0161] In an optional implementation, the confidence filtering module 204 is used to determine the preset temperature parameter by the following method: From the model validation dataset of the document classification model, obtain multiple sample documents and the true document category to which each sample document belongs; For each sample document, the sample document is input into the document classification model, and the document classification model predicts the document category to which the sample document belongs, thereby obtaining the original classification logical value corresponding to the sample document; For each of the multiple candidate temperature parameters, the original classification logic value corresponding to the sample document is divided by the candidate temperature parameter to obtain the target temperature scaling calibration result. The target temperature scaling calibration result is then normalized using the softmax function to obtain the calibration prediction probability of the sample document for the real document category under the candidate temperature parameter. Based on the calibration prediction probability corresponding to each sample document under the candidate temperature parameter, calculate the loss function value corresponding to the candidate temperature parameter; From the plurality of candidate temperature parameters, the candidate temperature parameter with the smallest loss function value is determined as the preset temperature parameter.

[0162] In an optional implementation, the multi-level confidence threshold includes at least a low confidence threshold, wherein, when determining the confidence category of each document in the second intermediate corpus based on the confidence calibration result and the preset multi-level confidence threshold, the confidence filtering module 204 is used to: For each document in the second intermediate corpus, determine whether the confidence calibration result corresponding to the document is less than or equal to the low confidence threshold; If the confidence calibration result corresponding to the document is less than or equal to the low confidence threshold, then the document is determined to belong to the low confidence category; wherein, the low confidence category is a confidence category that is lower than the preset standard.

[0163] In an optional implementation, after determining the confidence category to which each document in the second intermediate corpus belongs, the confidence filtering module 204 is further configured to: From the original classification logic values ​​corresponding to each document in the second intermediate corpus, determine the category prediction result corresponding to each document in the second intermediate corpus; wherein, the category prediction result is the preset document category corresponding to the largest prediction probability value among the original classification logic values; The category prediction results and the confidence category corresponding to each document in the second intermediate corpus are used as lightweight indexes for each document in the second intermediate corpus, and the lightweight indexes are transmitted to the central processing unit.

[0164] In an optional implementation, the confidence filtering module 204 is further configured to: The document category prediction and confidence calibration operations of the document classification model are performed on the second intermediate corpus of the current batch through the first CUDA stream. Simultaneously, the second intermediate corpus of the next batch is preprocessed by the central processing unit to obtain the target second intermediate corpus that conforms to the input format of the document classification prediction model. The target second intermediate corpus is then asynchronously transmitted to the document classification model as the model input data of the next batch via the second CUDA stream. The document classification model is located in the video memory of the graphics processor.

[0165] like Figure 3 As shown, this application provides an electronic device 300 for executing the corpus cleaning method of this application. The device includes a memory 301, a processor 302, and a computer program stored in the memory 301 and executable on the processor 302. The memory 301 and the processor 302 are connected via a bus for communication. When the processor 302 executes the computer program, it implements the steps of the corpus cleaning method described above.

[0166] Specifically, the memory 301 and processor 302 mentioned above can be general-purpose memory and processor, without any specific limitations. When the processor 302 runs the computer program stored in the memory 301, it can execute the above-mentioned corpus cleaning method.

[0167] Corresponding to the corpus cleaning method in this application, this application embodiment also provides a computer-readable storage medium storing a computer program, which is executed by a processor to perform the steps of the above-described corpus cleaning method.

[0168] Specifically, the storage medium can be a general-purpose storage medium, such as a removable disk or hard disk. When the computer program on the storage medium is run, it can perform the above-mentioned corpus cleaning method.

[0169] In the embodiments provided in this application, it should be understood that the disclosed systems and methods can be implemented in other ways. The system embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and there may be other division methods in actual implementation. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Additionally, the coupling or direct coupling or communication connection shown or discussed may be through some communication interface; the indirect coupling or communication connection between systems or units may be electrical, mechanical, or other forms.

[0170] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0171] In addition, the functional units in the embodiments provided in this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0172] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0173] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. In addition, the terms "first", "second", "third", etc. are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0174] Finally, it should be noted that the above-described embodiments are merely specific implementations of this application, used to illustrate the technical solutions of this application, and not to limit them. The protection scope of this application is not limited thereto. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some of the technical features, within the scope of the technology disclosed in this application; and these modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application. All should be covered within the protection scope of this application. Therefore, the protection scope of this application should be determined by the protection scope of the claims.

Claims

1. A corpus cleaning method, characterized in that, The corpus cleaning method includes: By performing a pre-filtering operation included in the coarse screening stage on the original corpus, texts that do not meet the preset quality conditions are filtered out from the original corpus, and a first intermediate corpus that meets the preset quality conditions is obtained. All documents in the first intermediate corpus are mapped to the video memory of the graphics processor. The hash values ​​of all words in all documents are calculated in parallel using multiple preset hash functions through a single kernel call of the graphics processor. The hash values ​​are then reduced to minimum values ​​along the word dimension in the video memory of the graphics processor to obtain the target signature matrix corresponding to the first intermediate corpus. Each row in the target signature matrix represents the target hash signature of each document in the first intermediate corpus. In the graphics processor's video memory, the target signature matrix is ​​subjected to local sensitive hashing bucketing to determine the duplicate documents contained in the first intermediate corpus; and based on the duplicate documents, the first intermediate corpus is deduplicated to obtain the deduplicated first intermediate corpus as the second intermediate corpus. The second intermediate corpus is input into a pre-trained document classification model, and the document classification model predicts the document category of each document in the second intermediate corpus to obtain the original classification logical value corresponding to each document in the second intermediate corpus; wherein, the original classification logical value is a vector composed of the unnormalized score output by the document classification model and corresponding to each preset document category; In the video memory of the graphics processor, the original classification logic value is calibrated with confidence to obtain the confidence calibration result corresponding to each document in the second intermediate corpus. Based on the confidence calibration result and the preset multi-level confidence threshold, the confidence category to which each document in the second intermediate corpus belongs is determined. From the second intermediate corpus, documents with confidence levels below a preset standard are filtered out, and the remaining documents are used as target corpus for training a large language model.

2. The corpus cleaning method according to claim 1, characterized in that, The preset pre-filtering operation includes at least one of the following: Filter out single text entries whose character count falls outside the preset character count range; Filter out individual texts whose language recognition results do not meet preset recognition conditions; wherein, the language recognition result is the probability value of the individual text belonging to the target language, which is input into the language recognition model and output by the language recognition model. Text standardization processing operations that convert a single text line into the target text format; Filter out individual texts whose text statistical features do not meet preset feature conditions; wherein, the text statistical features are determined based on at least one preset text statistical feature index; Filter out individual texts whose perplexity calculation result is higher than the preset perplexity threshold.

3. The corpus cleaning method according to claim 1, characterized in that, The step of mapping all documents in the first intermediate corpus to the graphics processor's video memory, and then using a single kernel call of the graphics processor to calculate the hash values ​​of all words in the documents under multiple preset hash functions in parallel, includes: The bytes of all the documents are concatenated into a continuous byte sequence, and the continuous byte sequence and the starting offset of each document in the continuous byte sequence are mapped to the video memory of the graphics processor. A two-dimensional thread grid is launched on the graphics processor. The two-dimensional thread grid is called by a single kernel to calculate the hash value of all the terms under each preset hash function in parallel, and the calculation result of the hash value is obtained. The first dimension of the two-dimensional thread grid corresponds to the document number of each document in the total number of documents, and the second dimension of the two-dimensional thread grid corresponds to the term position of each term in the document. Each thread block in the two-dimensional thread grid contains multiple threads corresponding to the multiple preset hash functions.

4. The corpus cleaning method according to claim 3, characterized in that, The calculation result of the hash value is reduced to a minimum value along the lexical dimension to obtain the target signature matrix corresponding to the first intermediate corpus, including: The calculation result of the hash value is written into a memory tensor of a first shape; wherein, the first shape is a multi-dimensional shape composed of the number of documents in all documents, the maximum number of words contained in a single document in all documents, and the number of preset hash functions; The memory tensor is reduced to a minimum along the lexical dimension corresponding to the maximum number of lexical units to obtain a signature matrix of a second shape as the target signature matrix; wherein, the second shape is a multidimensional shape composed of the number of documents in all documents and the number of preset hash functions.

5. The corpus cleaning method according to claim 4, characterized in that, The step of performing local sensitive hashing on the target signature matrix to determine duplicate documents contained in the first intermediate corpus includes: The target signature matrix is ​​divided into multiple strip sub-matrices along the column direction; wherein, the column direction is the dimension direction corresponding to the number of the preset hash functions in the target signature matrix, and the strip sub-matrices contain multiple consecutive columns of the target signature matrix; For each of the strip submatrixes, the row hash value is calculated for each row of the strip submatrix to obtain the strip hash value vector corresponding to the strip submatrix. Sort each of the strip hash value vectors according to the ascending or descending order of hash values ​​to obtain the sorting result; From the sorting results, documents with adjacent and equal hash values ​​are identified as the duplicate documents.

6. The corpus cleaning method according to claim 1, characterized in that, The step of calibrating the confidence level of the original classification logic values ​​to obtain the confidence level calibration results for each document in the second intermediate corpus includes: For each document in the second intermediate corpus, the original classification logic value corresponding to the document is divided by a preset temperature parameter to obtain the temperature scaling calibration result of the original classification logic value. The temperature scaling calibration result is normalized using the softmax function, and the normalized result is used as the confidence calibration result corresponding to the document.

7. The corpus cleaning method according to claim 6, characterized in that, The preset temperature parameter is determined using the following method: From the model validation dataset of the document classification model, obtain multiple sample documents and the true document category to which each sample document belongs; For each sample document, the sample document is input into the document classification model, and the document classification model predicts the document category to which the sample document belongs, thereby obtaining the original classification logical value corresponding to the sample document; For each of the multiple candidate temperature parameters, the original classification logic value corresponding to the sample document is divided by the candidate temperature parameter to obtain the target temperature scaling calibration result. The target temperature scaling calibration result is then normalized using the softmax function to obtain the calibration prediction probability of the sample document for the real document category under the candidate temperature parameter. Based on the calibration prediction probability corresponding to each sample document under the candidate temperature parameter, calculate the loss function value corresponding to the candidate temperature parameter; From the plurality of candidate temperature parameters, the candidate temperature parameter with the smallest loss function value is determined as the preset temperature parameter.

8. The corpus cleaning method according to claim 1, characterized in that, The multi-level confidence threshold includes at least a low confidence threshold, wherein determining the confidence category of each document in the second intermediate corpus based on the confidence calibration result and the preset multi-level confidence threshold includes: For each document in the second intermediate corpus, determine whether the confidence calibration result corresponding to the document is less than or equal to the low confidence threshold; If the confidence calibration result corresponding to the document is less than or equal to the low confidence threshold, then the document is determined to belong to the low confidence category; wherein, the low confidence category is a confidence category that is lower than the preset standard.

9. The corpus cleaning method according to claim 1, characterized in that, After determining the confidence category to which each document in the second intermediate corpus belongs, the corpus cleaning method further includes: From the original classification logic values ​​corresponding to each document in the second intermediate corpus, determine the category prediction result corresponding to each document in the second intermediate corpus; wherein, the category prediction result is the preset document category corresponding to the largest prediction probability value among the original classification logic values; The category prediction results and the confidence category corresponding to each document in the second intermediate corpus are used as lightweight indexes for each document in the second intermediate corpus, and the lightweight indexes are transmitted to the central processing unit.

10. The corpus cleaning method according to claim 1, characterized in that, The corpus cleaning method also includes: The document category prediction and confidence calibration operations of the document classification model are performed on the second intermediate corpus of the current batch through the first CUDA stream. Simultaneously, the second intermediate corpus of the next batch is preprocessed by the central processing unit to obtain the target second intermediate corpus that conforms to the input format of the document classification prediction model. The target second intermediate corpus is then asynchronously transmitted to the document classification model as the model input data of the next batch via the second CUDA stream. The document classification model is located in the video memory of the graphics processor.

11. A corpus cleaning device, characterized in that, The corpus cleaning device includes: The coarse screening module is used to filter out texts that do not meet the preset quality conditions from the original corpus by performing a preset pre-filtering operation included in the coarse screening stage, and obtain a first intermediate corpus that meets the preset quality conditions. The deduplication module maps all documents in the first intermediate corpus to the video memory of the graphics processor. It then uses a single kernel call of the graphics processor to calculate the hash values ​​of all terms in all documents using multiple preset hash functions in parallel. Finally, it performs minimum reduction on the hash values ​​along the term dimension in the video memory of the graphics processor to obtain the target signature matrix corresponding to the first intermediate corpus. Each row in the target signature matrix represents the target hash signature of each document in the entire corpus. In the graphics processor's video memory, the target signature matrix is ​​subjected to local sensitive hashing bucketing to determine the duplicate documents contained in the first intermediate corpus; and based on the duplicate documents, the first intermediate corpus is deduplicated to obtain the deduplicated first intermediate corpus as the second intermediate corpus. The classification module is used to input the second intermediate corpus into a pre-trained document classification model, predict the document category of each document in the second intermediate corpus through the document classification model, and obtain the original classification logical value corresponding to each document in the second intermediate corpus; wherein, the original classification logical value is a vector composed of the unnormalized score output by the document classification model and the corresponding preset document category; The confidence filtering module is used to perform confidence calibration on the original classification logic value in the graphics processor's video memory to obtain the confidence calibration result corresponding to each document in the second intermediate corpus, and determine the confidence category to which each document in the second intermediate corpus belongs based on the confidence calibration result and the preset multi-level confidence threshold. From the second intermediate corpus, documents with confidence levels below a preset standard are filtered out, and the remaining documents are used as target corpus for training a large language model.

12. An electronic device, characterized in that, include: The device includes a processor, a memory, and a bus. The memory stores machine-readable instructions executable by the processor. When the electronic device is running, the processor communicates with the memory via the bus. When the machine-readable instructions are executed by the processor, the steps of the corpus cleaning method as described in any one of claims 1 to 10 are performed.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, performs the steps of the corpus cleaning method as described in any one of claims 1 to 10.

Images