Method and system for constructing corpus based on webpage data set

Abstract

According to the method and system for constructing the corpus based on the webpage data set, the large-scale and high-quality corpus can be constructed based on the massive webpage data set through a series of operations of mass filtering, weight removal, hatred speech detection and confusion degree calculation; the steps of quality filtering and data deduplication greatly reduce repeated and low-quality contents in a corpus, and improve the generalization ability of the model and the reliability of output; in addition, high-standard and multi-level quality of the corpus is further ensured through hatred speech detection and puzzle calculation, potential negative effects are reduced, and the model is more accurate and efficient when processing various natural language tasks.

Description

Technical Field

[0001] The present invention belongs to the technical field of data processing, and in particular relates to a method and system for constructing a corpus based on a web page data set. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] At present, with the rapid development of large language models such as ChatGPT, YUAN1.0, and Pangu α, their applications and influence are rapidly expanding, showing excellent ability to process and generate language. As the parameter scale of these models continues to grow, the demand for large-scale and high-quality corpora is also increasing. As one of the important sources of corpora, Internet web pages continue to generate a large amount of new content every day. This not only provides a rich data resource for the training of language models, but also brings the challenge of how to efficiently extract and process these data.

[0004] There are two main necessities for building a high-quality corpus based on web page datasets. First, for large pre-trained language models, it is crucial to ensure the accuracy and high quality of the data. High-quality corpora can improve the generalization ability of the model and reduce the risk of bias, thereby ensuring the reliability and practicality of the model output. Secondly, massive web page datasets contain a large amount of repetitive, indecent and low-quality content, and the text quality is uneven, which cannot be directly applied to pre-trained language models. Simply increasing the size of the corpus does not necessarily lead to an improvement in model performance. Ensuring the quality of the corpus has become the key to model optimization.

[0005] Therefore, how to build a large-scale, high-quality corpus based on massive web page data sets is a technical problem that needs to be solved. Summary of the invention

[0006] In order to overcome the shortcomings of the above-mentioned prior art, the present invention provides a method and system for constructing a corpus based on a web page dataset. Through a series of operations such as quality filtering, deduplication, hate speech detection and perplexity calculation, a large-scale, high-quality corpus is constructed based on a massive web page dataset.

[0007] In order to achieve the above object, the present invention adopts the following technical solution:

[0008] In a first aspect, the present invention provides a method for constructing a corpus based on a web page dataset, comprising:

[0009] Perform text extraction on the web page data in the acquired original text, save each web page as a document, and perform quality filtering operations on each document;

[0010] Remove duplicates from the filtered documents, split similar documents into multiple independent sentences, form sentence groups from consecutive independent sentences, perform word segmentation on each independent sentence, combine each word in the sentence group with sentence information and position information to form a vector representation of each sentence group, calculate the similarity between the vector representations of each pair of sentence groups, and delete duplicate sentence groups;

[0011] Detect offensive speech in each document based on the trained language model and delete sentences containing offensive speech;

[0012] The perplexity score is calculated for each document based on the trained perplexity calculation model, and a corpus is constructed based on the perplexity score.

[0013] In a second aspect, the present invention provides a system for constructing a corpus based on a web page dataset, comprising:

[0014] A quality filtering module is configured to: extract text from web page data in the acquired original text, save each web page as a document, and perform a quality filtering operation on each document;

[0015] A data deduplication module is configured to: remove duplicates from the filtered documents, split similar documents into multiple independent sentences, form sentence groups from consecutive independent sentences, perform word segmentation on each independent sentence, form a vector representation of each sentence group by combining each word in the sentence group with sentence information and position information, calculate the similarity between the vector representations of each sentence group, and delete duplicate sentence groups;

[0016] An offensive speech detection module, which is configured to: detect offensive speech for each document based on the trained language model, and delete sentences containing offensive speech;

[0017] The perplexity calculation module is configured to calculate the perplexity score for each document based on the trained perplexity calculation model, and construct a corpus according to the perplexity score.

[0018] In a third aspect, the present invention provides an electronic device comprising a memory and a processor, and computer instructions stored in the memory and executed on the processor, wherein when the computer instructions are executed by the processor, the method described in the first aspect is performed.

[0019] In a fourth aspect, the present invention provides a computer-readable storage medium for storing computer instructions, wherein when the computer instructions are executed by a processor, the method described in the first aspect is performed.

[0020] One or more of the above technical solutions have the following beneficial effects:

[0021] In the present invention, through a series of operations including quality filtering, deduplication, hate speech detection and perplexity calculation, a large-scale, high-quality corpus can be constructed based on a massive web page data set; the quality filtering and data deduplication steps significantly reduce the repetitive and low-quality content in the corpus, and improve the generalization ability of the model and the reliability of the output; in addition, hate speech detection and perplexity calculation further ensure the high standards and multi-level quality of the corpus, reduce potential negative effects, and make the model more accurate and efficient when processing various natural language tasks.

[0022] Advantages of additional aspects of the present invention will be given in part in the following description, and in part will become obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0024] Figure 1 This is an overall flow chart of building a high-quality corpus based on a web page dataset in Embodiment 1 of the present invention;

[0025] Figure 2 This is a schematic diagram of the language recognition process in the first embodiment of the present invention;

[0026] Figure 3 It is a schematic diagram of the quality filtering function flow in the first embodiment of the present invention;

[0027] Figure 4 Schematic diagram of sentence group vector generation by sentence group in Embodiment 1 of the present invention;

[0028] Figure 5 It is a schematic diagram of the hate speech detection function flow in the first embodiment of the present invention. DETAILED DESCRIPTION

[0029] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0030] It should be noted that the terms used herein are for describing specific embodiments only and are not intended to be limiting of exemplary embodiments according to the present invention.

[0031] In the absence of conflict, the embodiments of the present invention and the features of the embodiments may be combined with each other.

[0032] Embodiment 1

[0033] This embodiment discloses a method for constructing a corpus based on a web page dataset, comprising:

[0034] Perform text extraction on the web page data in the acquired original text, save each web page as a document, and perform quality filtering operations on each document;

[0035] Remove duplicates from the filtered documents, split similar documents into multiple independent sentences, form sentence groups from consecutive independent sentences, perform word segmentation on each independent sentence, combine each word in the sentence group with sentence information and position information to form a vector representation of each sentence group, calculate the similarity between the vector representations of each pair of sentence groups, and delete duplicate sentence groups;

[0036] Detect offensiveness of documents based on the trained language model and delete sentences containing offensive remarks;

[0037] The perplexity score is calculated for each document based on the trained perplexity calculation model, and a corpus is constructed based on the perplexity score.

[0038] This embodiment efficiently constructs a large-scale, high-quality corpus from a web page dataset through a pipeline method. This method is highly flexible and can be easily expanded to multiple languages ​​with simple adjustments. The core processing steps of the pipeline include: language identification to distinguish and select content in the target language; quality filtering to remove low-quality and irrelevant information; data deduplication to reduce the repetitiveness of corpus text and improve the generalization ability of the model; hate speech detection to maintain the quality of the corpus while reducing potential negative effects; and perplexity calculation to remove documents with too high perplexity and divide the corpus into three different levels. Through a series of processing, a large-scale, high-quality corpus can be provided for a large pre-trained language model.

[0039] This embodiment combines Figure 1 Taking reading the warc or warc.gz file provided by the Common Crawl dataset as an example, the language recognition, quality filtering, data deduplication, hate speech detection, and perplexity calculation involved are explained in detail.

[0040] In this embodiment, language identification: a text extraction tool is used to extract the web page text content from all web page data in the input original file, a language filter is used to identify the required language web pages with a score greater than 0.65 for the extracted web page text content, and each web page is saved as a JSON object, i.e., a document.

[0041] Specifically, Figure 2As shown, read all warc or warc.gz files in the folder, process each record in the file, that is, each web page document, and obtain the web page content text, web page ID, and web page URL of each web page. Use the text extraction tool Trafilatura to extract the web page body content for each web page, and then use the language filter Fasttext to perform language recognition on the body text to filter out Chinese web pages with a score of more than 0.65; after processing, save each web page that meets the standard in the following JSON object format, and save each warc or warc.gz file as a JSON file.

[0042] The JSON object format is as follows:

[0043]

[0044]

[0045] In this embodiment, quality filtering: each JSON object is processed. Specifically, documents containing banned words are deleted.

[0046] Specific processing is performed on the line-level content: the content before the sentence where the first punctuation mark is located and the content after the last punctuation mark are deleted;

[0047] Delete illegal characters such as [\x00-\x08\x0B-\x0C\x0E-\x1F\x7F\u00A0\u2028\u2029\u3000] on web pages; remove documents containing "lorem ipsum"; delete documents with an average of 5 characters per line; delete lines that start with a bullet or end with an ellipsis more than the specified requirement; remove lines containing "javascript" and lines containing strings related to cookies.

[0048] Specifically, Figure 3 As shown, determine whether the URL is in the blacklist. If not, first divide the line according to common punctuation marks or line breaks, and make the following judgments on the whole and each line: determine whether the ratio of line breaks to the number of words in the JSON object is greater than a set value, such as 0.2. If so, delete the JSON object; determine whether the proportion of lines starting with bullet points exceeds a set value, such as 0.9. If so, delete the JSON object; determine whether the proportion of lines ending with ellipsis exceeds a set value, such as 0.4. If so, delete the JSON object; determine whether there is javascript in the line. If so, delete the line.

[0049] Provide a list of banned words, determine whether the document contains the words in the list multiple times, if it appears 5 times or more, delete the JSON object; determine whether the number of sentences in the document is less than 5, if so, delete the JSON object; delete the content before the sentence where the first tag symbol is located and after the last punctuation mark. Finally, limit the text length of the processed JSON object to between 50 and 100,000 words, if so, keep the JSON object.

[0050] In this embodiment, data deduplication: the documents after quality filtering are deduplicated at the URL level and sentence group level. URL deduplication determines whether the URLs of the documents are equal. If they are equal, the document is deleted based on the document priority. The document priority here is determined based on the document length.

[0051] Deduplication at the sentence group level: perform sentence group level similarity comparison on documents. First, find documents with high similarity and group them into a group. Perform sentence group level similarity comparison on the documents in the group and keep only one of all duplicates.

[0052] Specifically, first, convert the URL addresses of all JSON objects to lowercase to unify the format; then, generate a hash value for each URL address, and combine this hash value with the priority of the JSON object and the ID of the JSON object to form a unique signature. The priority of the JSON object here refers to the length of the text; then, scatter these signatures into multiple files, and sort the signatures in each file according to the hash value.

[0053] Build a priority queue: First, extract the first signature from each file and add it to the priority queue; next, compare the hash values ​​of the signatures in the priority queue to identify duplicates. If duplicate document signatures are found, keep the JSON object with high priority and delete the object with low priority. If no duplicates are found, continue to take the next signature from each JSON file and add it to the priority queue; repeat this process until all signatures have been processed.

[0054] The sentence group level deduplication function of the data deduplication function first uses the local sensitive hashing method to compare the similarity of all documents, clusters all similar documents into the same bucket, and then compares the similarity of each bucket at the sentence group level.

[0055] like Figure 4As shown, the document in each bucket is segmented using common punctuation marks (.?!....;) and split into multiple independent sentences. For each independent sentence, the sentence ID and document ID are recorded. Group n consecutive sentences together, where n is set to 2, and call them sentence groups. For example, for four sentences A, B, C, and D, AB is a sentence group and CD is another sentence group. When the number of remaining sentences in the document is less than n, they are grouped separately.

[0056] First, each sentence is segmented, and then punctuation marks and words without core meaning (such as, de, di, le, zhe, guo, ne, ma, ba, ya, ah, la, ma, just, very, very, special, extremely, very, slightly, gradually, just now, always, often, only, almost, almost, as expected, actually, actually, in, to, to, from, for, by, because, this, that, these, those, etc.) are removed to reduce their influence on similarity judgment. Then the vector of each remaining word in the sentence group is calculated and combined with the sentence information, i.e., the sentence number of the sentence group, and the position information, i.e., the position of each word in the sentence group, to form a vector representation of each sentence group.

[0057] In each bucket, first randomly select a document and compare the sentence groups in it with the sentence groups in another document in the bucket one by one, and calculate the Euclidean distance between any two sentence group vectors as the basis for measuring the similarity of the sentence groups. In this process, record all the information of sentence groups that are judged to be similar. Subsequently, this document continues to perform the same sentence group comparison with the remaining documents in the bucket that have not yet participated in the comparison until all comparisons between any two documents in the bucket are completed. For similar sentence groups, only one of them is retained to avoid content duplication. Finally, the starting and ending positions of each sentence are calculated based on the document ID and sentence ID, and these duplicate sentences are deleted from the original document.

[0058] In this embodiment, hate speech detection: Detect and delete documents that repeatedly contain offensive speech. Collect multiple public offensive speech data sets, and use these data sets to train the pre-trained language model ERNIE to detect offensive speech. Perform offensive detection on each sentence, and delete the document if offensive speech appears repeatedly in the document.

[0059] Specifically, Figure 5As shown in the figure, we collect data sets of offensive speech, including data sets of sexual and antisocial offensive speech published by Tang et al. and data sets of speech related to social prejudice published by Yang et al., and integrate these data sets into one set. We use this set to train an offensive speech detector based on the pre-trained language model ERNIE for Chinese. First, we mark the sorted set to ensure that each label has a corresponding label. The labels are divided into: 0 for no hate speech, 1 for attacking a specific person, 2 for attacking a certain group, 3 for antisocial and related to social prejudice, and 4 for sex. Then we integrate them into a data set of offensive speech. This process needs to ensure the balance of each type of speech to avoid the subsequent training model from being biased towards one type.

[0060] Before training, each text is processed, including processing stop words and label symbols in the text to reduce their impact on the results. Use a tokenizer to convert each text into input_ids and attention_mask. During training, the framework's DataLoader is used to divide the data into multiple batches, and the cross entropy loss function is used for training, and multiple rounds of iterations are performed to ensure the accuracy of the results.

[0062] Since the resource requirements for detecting hate speech in every sentence of a large number of documents are too high, the method of extracting the key sentences of each paragraph of text for detection is adopted. For each document, the line break is used to distinguish whether the text is segmented. For each paragraph of text, sentence segmentation and stop word processing are first performed, and then a graph structure is constructed based on the similarity between sentences (TF-IDF value), in which each sentence is a node of the graph and the similarity is used as the weight of the edge between nodes. Next, by simulating the random walk process of particles on the graph, the weight of each sentence is iteratively updated. The weight update is based on the weight of its adjacent sentences and the similarity between them. When the weight change tends to be stable or reaches the preset number of iterations, the iteration is stopped. Finally, the sentences are sorted according to their final weights, and the three sentences with the highest weights are selected as key sentences. These sentences can represent the main content and core ideas of the text.

[0063] The trained model is used to detect offensive speech in the extracted key sentences. The model will generate an offensive classification for each key sentence. The paragraphs with output results of 1 or 4 that appear twice or more will be deleted, and the paragraphs containing key sentences with output results of 2 or 3 will be directly deleted.

[0064] In this embodiment, perplexity calculation: a perplexity score is generated for the processed document. The lower the score, the higher the quality of the document. This process uses a 5-gram Kneser-Ney language model based on the KenLM library. The model is trained with public high-quality content such as news and Wikipedia data sets. The perplexity of each document is calculated based on the trained model. The document is divided into three parts according to the perplexity score.

[0065] Specifically, select high-quality data sets from multiple sources, such as news covering politics, economy, technology, sports and other fields, as well as Wikipedia, and perform appropriate preprocessing (remove HTML tags, normalize punctuation, and convert uppercase English to lowercase). For Chinese, it is necessary to perform word segmentation on the content and divide the text into a sequence of words or tokens. Subsequently, the language model is trained using the KenLM library, and 5-gram is specified as the order of n-gram. The shared information of 5 consecutive words is considered to capture the complex relationship and context dependency. The Kneser-Ney smoothing technique is used to deal with zero-frequency or low-frequency vocabulary problems in the training data to improve the model's prediction ability for unseen words. After the training is completed, a 5-gram Kneser-Ney language model implemented by the KenLM library will be obtained. The perplexity of each document is calculated using the trained language model. The higher the perplexity, the lower the similarity between the document and the adopted high-quality text, but it cannot be directly considered as the quality of its content. Therefore, the document is evenly divided into three parts: head, middle, and tail according to the perplexity score, which is convenient for the subsequent training of large language models.

[0066] The parallel processing function is implemented using Slurm. Slurm clusters are deployed on multiple nodes and are suitable for a variety of tasks, including key processes such as language recognition, quality filtering, hate speech detection, and perplexity calculation. Each step is processed in parallel, that is, the data set can be divided into different data blocks for separate processing. The use of Slurm optimizes resource allocation and improves the efficiency and flexibility of job execution. It is particularly suitable for complex processing environments that need to process large amounts of data. In addition, the system supports highly configurable job scheduling and flexible multi-node task processing capabilities, ensuring efficient and stable operation performance.

[0067] The parallel processing function is implemented using Slurm, which is specifically designed to efficiently manage and coordinate job execution in a distributed computing environment. It can not only assign priorities to jobs and allocate resources, but also support multi-node tasks, making it particularly suitable for processing large-scale data and performing complex computing tasks. Language recognition, quality filtering, hate speech detection, and perplexity calculation can all be achieved through this system to achieve efficient parallel processing of data.

[0068] This embodiment constructs a large-scale, high-quality corpus from a web page data set, which significantly improves the training effect and practicality of the language model. Through precise language recognition, this embodiment can effectively distinguish and select target language content, ensuring the accuracy of the data. At the same time, through quality filtering and data deduplication steps, the repetitive and low-quality content in the corpus is greatly reduced, and the generalization ability of the model and the reliability of the output are improved. In addition, hate speech detection and perplexity calculation further ensure the high standards and multi-level quality of the corpus, reduce potential negative effects, and make the model more accurate and efficient when processing various natural language tasks. The flexible design of this embodiment also supports multi-language expansion, making it suitable for large-scale language model training needs in different languages ​​around the world. In short, this embodiment not only optimizes the construction process of the corpus, but also significantly improves the efficiency of data processing through parallel processing technology, providing a solid data foundation for the development of language models.

[0069] Embodiment 2

[0070] The purpose of this embodiment is to provide a system for constructing a corpus based on a web page dataset, including:

[0071] A quality filtering module is configured to: extract text from web page data in the acquired original text, save each web page as a document, and perform a quality filtering operation on each document;

[0072] A data deduplication module is configured to: remove duplicates from the filtered documents, split similar documents into multiple independent sentences, form sentence groups from consecutive independent sentences, perform word segmentation on each independent sentence, form a vector representation of each sentence group by combining each word in the sentence group with sentence information and position information, calculate the similarity between the vector representations of each sentence group, and delete duplicate sentence groups;

[0073] An offensive speech detection module, which is configured to: detect offensive speech in a document based on a trained language model, and delete sentences containing offensive speech;

[0074] The perplexity calculation module is configured to calculate the perplexity score for each document based on the trained perplexity calculation model, and construct a corpus according to the perplexity score.

[0075] In further embodiments, there is also provided:

[0076] An electronic device includes a memory and a processor, and computer instructions stored in the memory and executed on the processor, wherein when the computer instructions are executed by the processor, the method described in Embodiment 1 is performed. For the sake of brevity, no further description is given here.

[0077] It should be understood that in this embodiment, the processor may be a central processing unit CPU, and the processor may also be other general-purpose processors, digital signal processors DSP, application-specific integrated circuits ASIC, off-the-shelf programmable gate arrays FPGA or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0078] The memory may include a read-only memory and a random access memory, and provide instructions and data to the processor. A portion of the memory may also include a non-volatile random access memory. For example, the memory may also store information about the device type.

[0079] A computer-readable storage medium is used to store computer instructions. When the computer instructions are executed by a processor, the method described in embodiment 1 is completed.

[0080] The method in the first embodiment can be directly embodied as a hardware processor, or a combination of hardware and software modules in the processor. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware. To avoid repetition, it will not be described in detail here.

[0081] A computer program product includes a computer program, and when the computer program is executed by a processor, the method described in the first embodiment is implemented.

[0082] The present invention also provides at least one computer program product tangibly stored on a non-transitory computer-readable storage medium. The computer program product includes computer executable instructions, such as instructions included in a program module, which are executed in a device on a real or virtual processor of the target to perform the process / method as described above. Typically, program modules include routines, programs, libraries, objects, classes, components, data structures, etc. that perform specific tasks or implement specific abstract data types. In various embodiments, the functions of program modules can be combined or divided between program modules as needed. Machine executable instructions for program modules can be executed in local or distributed devices. In distributed devices, program modules can be located in local and remote storage media.

[0083] The computer program code for implementing the method of the present invention can be written in one or more programming languages. These computer program codes can be provided to the processor of a general-purpose computer, a special-purpose computer or other programmable data processing device, so that the program code, when executed by the computer or other programmable data processing device, causes the function / operation specified in the flow chart and / or block diagram to be implemented. The program code can be executed completely on a computer, partially on a computer, as an independent software package, partially on a computer and partially on a remote computer or completely on a remote computer or server.

[0084] In the context of the present invention, computer program codes or related data may be carried by any appropriate carrier to enable a device, apparatus or processor to perform the various processes and operations described above. Examples of carriers include signals, computer readable media, etc. Examples of signals may include electrical, optical, radio, acoustic or other forms of propagation signals, such as carrier waves, infrared signals, etc.

[0085] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with this embodiment can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0086] Although the above describes the specific implementation mode of the present invention in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art on the basis of the technical solution of the present invention without creative work are still within the scope of protection of the present invention.

Claims

1. A method for constructing a corpus based on a web page dataset, characterized in that: include: Perform text extraction on the web page data in the acquired original text, save each web page as a document, and perform quality filtering operations on each document; Remove duplicates from the filtered documents, split similar documents into multiple independent sentences, form sentence groups from consecutive independent sentences, perform word segmentation on each independent sentence, combine each word in the sentence group with sentence information and position information to form a vector representation of each sentence group, calculate the similarity between the vector representations of each pair of sentence groups, and delete duplicate sentence groups; Detect offensiveness of documents based on the trained language model and delete sentences containing offensive remarks; The perplexity score is calculated for each document based on the trained perplexity calculation model, and a corpus is constructed based on the perplexity score.

2. A method for constructing a corpus based on a web page dataset as claimed in claim 1, characterized in that: Perform text extraction on the web page data in the acquired original text and save each web page as a document, specifically: Get the webpage content text, webpage ID, and webpage URL of each webpage; For each web page, a text extraction tool is used to extract the main text of the web page, and a language filter is used to identify the language of the main text. After the web pages in the target language are screened out, they are saved in the form of JSON objects.

3. The method for constructing a corpus based on a web page dataset as claimed in claim 1, characterized in that: Perform quality filtering operations on each document, specifically: Determine whether the webpage URL is in the blacklist. If not, split it into lines according to common punctuation marks or line breaks; Determine whether the ratio of line breaks to word count in the JSON object is greater than the corresponding set value. If so, delete the corresponding JSON object. Determine whether the proportion of rows starting with a bullet point exceeds the corresponding set value. If so, delete the corresponding JSON object; Determine whether the proportion of rows ending with ellipsis exceeds the corresponding set value. If so, delete the corresponding JSON object; Determine whether there is javascript in the line, if so, delete the corresponding line.

4. The method for constructing a corpus based on a web page dataset according to claim 1, characterized in that: Remove duplicates from filtered documents based on URL web pages, specifically: Calculate the hash value of the webpage URL of each JSON object, and combine the calculated hash value with the priority and ID of the corresponding JSON object into a signature; Spread the signatures into multiple files and sort the signatures by hash value within each file; Extract the first signature from each file and add it to the priority queue; Identify the signatures in the priority queue, retain the JSON objects with high priority for the duplicate signatures, and delete the JSON objects with low priority; If there are no duplicates in the priority queue, continue to extract the next signature from each file and add it to the priority queue, repeating the above process until all signatures are processed.

5. The method for constructing a corpus based on a web page dataset according to claim 1, characterized in that: Before splitting similar documents into multiple independent sentences, the local sensitive hashing method is used to compare the similarity of all documents, cluster all similar documents into the same bucket, and perform similarity comparison at the sentence group level for each bucket.

6. A method for constructing a corpus based on a web page dataset as claimed in claim 5, characterized in that: In each bucket, a document is randomly selected, and the sentence groups in the document are compared with the sentence groups in other documents in the bucket. The Euclidean distance between the two sentence group vectors is calculated to determine the similarity of the sentence groups. For similar sentence groups, one of the sentence groups is deleted.

7. The method for constructing a corpus based on a web page dataset according to claim 1, characterized in that: Based on the trained language model, the sentence groups are tested for offensiveness and sentences containing offensive remarks are deleted. Specifically: Perform sentence processing on the text field in the JSON object; Add a special marker in front of each sentence after the clause; Each sentence with a special tag is used to identify offensive speech based on the trained language model; Count the number of times each JSON object contains offensive speech, and delete the corresponding JSON objects that exceed the set number.

8. A system for constructing a corpus based on a web page dataset, characterized in that: include: A quality filtering module is configured to: extract text from web page data in the acquired original text, save each web page as a document, and perform a quality filtering operation on each document; A data deduplication module is configured to: remove duplicates from the filtered documents, split similar documents into multiple independent sentences, form sentence groups from consecutive independent sentences, perform word segmentation on each independent sentence, form a vector representation of each sentence group by combining each word in the sentence group with sentence information and position information, calculate the similarity between the vector representations of each sentence group, and delete duplicate sentence groups; An offensive speech detection module, which is configured to: detect offensive speech in a document based on a trained language model, and delete sentences containing offensive speech; The perplexity calculation module is configured to calculate the perplexity score for each document based on the trained perplexity calculation model, and construct a corpus according to the perplexity score.

9. An electronic device, characterized in that: The method comprises a memory and a processor and computer instructions stored in the memory and executed on the processor, wherein when the computer instructions are executed by the processor, the method according to any one of claims 1 to 7 is completed.

10. A computer-readable storage medium, characterized in that: Used to store computer instructions, which, when executed by a processor, complete the method described in any one of claims 1 to 7.

Images