Information processing method, device and equipment and computer storage medium
By screening target information, processing quality classification model and optimizing data on corpus data, the problem of poor quality of Chinese corpus in the existing technology is solved, and the acquisition and improvement of high-quality corpus is achieved.
Patent Information
- Application Number
- CN202311128590.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-08-30
- Publication Date
- 2025-03-11
AI Technical Summary
The problem of poor Chinese corpus quality cannot be effectively solved in the prior art, especially in the pre-trained corpus of open source large models, the proportion of Chinese corpus is small, the quality of general-field corpus is uneven, and high-quality vertical corpus is lacking.
By screening the corpus data to be processed based on the target information, the target quality classification model is used to process the screened corpus data, and optimize the data characteristics of the corpus data to realize multiple rounds of screening and filtration to improve the corpus quality.
The quality of the corpus is significantly improved, and the problem of poor corpus quality cannot be effectively solved in related technologies, ensuring the high quality of the obtained target corpus data.
Smart Images
Figure CN119597736A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to information processing technology in the field of computer technology, and in particular to an information processing method, device, equipment and computer storage medium. Background Art
[0002] At present, the main corpus of the pre-training corpus of open source large models (such as LLaMA) is mostly in English, with a small proportion of Chinese corpus; the general domain corpus is mostly derived from the Internet corpus of web crawlers, with uneven quality, semantic incoherence, and contains a lot of repeated invalid content, such as advertisements, disclaimers, website navigation, directories, etc.; there is an extreme lack of high-quality corpus in vertical fields (such as network security, general computer knowledge, etc.). For the above corpus, there is no method for cleaning Chinese corpus, nor is there a method for obtaining high-quality corpus; and the relevant technology generally uses keywords, rule filtering and other methods to clean the corpus, but it cannot effectively solve the problem of poor corpus quality. Summary of the invention
[0003] In order to solve the above technical problems, the embodiments of the present application hope to provide an information processing method, device, equipment and computer storage medium, which solves the problem that the cleaning solution for corpus in the related technology cannot effectively solve the poor quality of corpus, and greatly improves the quality of corpus.
[0004] The technical solution of this application is implemented as follows:
[0005] An information processing method, the method comprising:
[0006] Based on the target information, the corpus data to be processed is screened to obtain screened corpus data; wherein the target information at least includes information related to the user's information and target words;
[0007] Using a target quality classification model to process the screened corpus data to obtain high-quality corpus data to be optimized;
[0008] The high-quality corpus data to be optimized is optimized based on the data features of the corpus data to obtain target corpus data.
[0009] In the above scheme, the filtering of the corpus data to be processed based on the target information to obtain the filtered corpus data includes:
[0010] Processing the to-be-processed corpus data based on the target data style to obtain processed corpus data;
[0011] The target information is removed from the processed corpus data to obtain the filtered corpus data.
[0012] In the above solution, the processing of the to-be-processed corpus data based on the target data style to obtain the processed corpus data includes:
[0013] Based on the target data pattern, determining target characters among the characters of the corpus data to be processed;
[0014] Processing the target characters in the corpus data to be processed to obtain the processed corpus data;
[0015] Correspondingly, removing the target information from the processed corpus data to obtain the filtered corpus data includes:
[0016] The target information in the processed corpus data is removed by using a target screening technique to obtain the screened corpus data.
[0017] In the above scheme, the method further includes:
[0018] Determine high-quality sample corpus data and low-quality sample corpus data;
[0019] Based on the high-quality sample corpus data and the low-quality sample corpus data, a binary classification model training is performed on the target pre-trained language representation model to obtain the target quality classification model; wherein the target quality classification model is capable of dividing the corpus data into high-quality corpus data and low-quality corpus data; the low-quality corpus data at least includes corpus data of invalid content, biased content, and semantically incoherent types.
[0020] In the above solution, the step of determining high-quality sample corpus data and low-quality sample corpus data includes:
[0021] Acquire the first sub-high-quality sample corpus data from the target high-quality corpus database;
[0022] Determine the sample corpus data to be classified;
[0023] Screening and processing the sample corpus data to be classified to obtain the low-quality sample corpus data and a second sub-high-quality sample corpus data;
[0024] The high-quality sample corpus data is determined based on the first sub-high-quality sample corpus data and the second sub-high-quality sample corpus data.
[0025] In the above scheme, the optimizing process of the high-quality corpus data to be optimized based on the data features of the corpus data to obtain the target corpus data includes:
[0026] Filtering the high-quality corpus data to be optimized based on the data features to obtain filtered corpus data;
[0027] The filtered corpus data is deduplicated to obtain the target corpus data.
[0028] In the above scheme, filtering the high-quality corpus data to be optimized based on the data features to obtain filtered corpus data includes:
[0029] Determining a target filtering condition based on the data characteristics;
[0030] The high-quality corpus data to be optimized is filtered based on the target filtering condition to obtain the filtered corpus data.
[0031] In the above scheme, the step of performing deduplication processing on the filtered corpus data to obtain the target corpus data includes:
[0032] Performing deduplication processing on the filtered corpus data based on the target substring to obtain the target corpus data;
[0033] and / or, determining the similarity between the filtered corpus data, and performing deduplication processing on the filtered corpus data based on the similarity to obtain the target corpus data;
[0034] And / or, deleting sentences whose semantic similarity meets the target similarity in the filtered corpus data to obtain the target corpus data.
[0035] An information processing device, comprising:
[0036] A screening unit, configured to screen the corpus data to be processed based on target information to obtain screened corpus data; wherein the target information at least includes information related to the user's private information and target words;
[0037] A processing unit, used to process the screened corpus data using a target quality classification model to obtain high-quality corpus data to be optimized;
[0038] The optimization unit is used to optimize the high-quality corpus data to be optimized based on the data features of the corpus data to obtain target corpus data.
[0039] An information processing device, the device comprising: a processor, a memory and a communication bus;
[0040] The communication bus is used to realize the communication connection between the processor and the memory;
[0041] The processor is used to execute the information processing program in the memory to implement the steps of the above-mentioned information processing method.
[0042] A computer-readable storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps of the above-mentioned information processing method.
[0043] The information processing method, apparatus, device and computer storage medium provided in the embodiments of the present application can screen and process the corpus data to be processed based on the target information to obtain the screened corpus data, wherein the target information at least includes information related to the user's information and target words, and the screened corpus data is processed using a target quality classification model to obtain high-quality corpus data to be optimized, and the high-quality corpus data to be optimized is optimized based on the data features of the corpus data to obtain the target corpus data. In this way, the corpus data can be screened based on the target information first, and then the screened corpus data can be further screened using the target quality classification model, and then further filtered according to the data features of the corpus data, that is, the corpus data can be screened for multiple rounds using the target information, the target quality classification model and the data features of the corpus data, thereby ensuring the quality of the obtained target corpus data, thereby solving the problem that the corpus cleaning solution in the related art cannot effectively solve the poor corpus quality, and greatly improving the quality of the corpus. BRIEF DESCRIPTION OF THE DRAWINGS
[0044] Figure 1 A flowchart of an information processing method provided in an embodiment of the present application;
[0045] Figure 2 A flowchart of another information processing method provided in an embodiment of the present application;
[0046] Figure 3 A schematic diagram of determining a target quality classification model in an information processing method provided in an embodiment of the present application;
[0047] Figure 4 A flowchart of another information processing method provided in an embodiment of the present application;
[0048] Figure 5 A schematic diagram of the structure of an information processing device provided in an embodiment of the present application;
[0049] Figure 6 A schematic diagram of the structure of an information processing device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0050] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.
[0051] It should be understood that the "embodiments of the present application" or "the aforementioned embodiments" mentioned throughout the specification mean that specific features, structures or characteristics related to the embodiments are included in at least one embodiment of the present application. Therefore, "in the embodiments of the present application" or "in the aforementioned embodiments" appearing throughout the specification may not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics may be combined in one or more embodiments in any suitable manner. In the various embodiments of the present application, the size of the sequence numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application. The above-mentioned sequence numbers of the embodiments of the present application are for description only and do not represent the advantages and disadvantages of the embodiments.
[0052] Unless otherwise specified, any step in the embodiments of the present application may be performed by a processor of the device. It is also worth noting that the embodiments of the present application do not limit the order in which the device executes the following steps. In addition, the methods used to process data in different embodiments may be the same method or different methods. It should also be noted that any step in the embodiments of the present application can be independently executed by the device, that is, when the device executes any step in the following embodiments, it may not rely on the execution of other steps.
[0053] It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.
[0054] The present application provides an information processing method, which can be applied to an information processing device. Figure 1 As shown, the method may include the following steps:
[0055] Step 101: Screen the corpus data to be processed based on target information to obtain screened corpus data.
[0056] The target information at least includes information related to the user's information and target words.
[0057] In an embodiment of the present application, the corpus data to be processed may refer to the corpus that needs to be screened and has low quality; in a feasible implementation, the corpus data to be processed may refer to the Chinese pre-trained corpus. The target information may refer to the information pre-set for sentence screening of the corpus data to be processed; wherein, the information related to the user's information may refer to some information of a company or an individual; in a feasible implementation, the information related to the user's information may include at least one of the following: email, uniform resource locator (URL), Internet Protocol (IP) address, ID number, instant messaging tool number, Hypertext Markup Language (HTML) tag, telephone number, company information, etc.; wherein, company information may refer to information such as company name. The target words may refer to words such as banned words and sensitive words.
[0058] It should be noted that the filtered corpus data may be obtained by first performing an initial processing on the corpus data to be processed, and then further filtering the corpus data to be processed after the initial processing according to the target information, so as to obtain the filtered corpus data.
[0059] Step 102: Use the target quality classification model to process the screened corpus data to obtain high-quality corpus data to be optimized.
[0060] The target quality classification model may refer to a model that has the function of classifying corpus data into high-quality corpus data and low-quality corpus data; and the target quality classification model may be obtained after model training with high-quality sample corpus data and low-quality sample corpus data. The high-quality corpus data to be optimized may refer to the corpus data with higher quality that is screened and processed from the screened corpus data after being processed by the target quality classification model.
[0061] In other embodiments of the present application, the filtered corpus data can be input as input parameters into the target quality classification model, and then the target quality classification model processes the filtered corpus data to output high-quality corpus data to be optimized.
[0062] Step 103: Optimize the high-quality corpus data to be optimized based on the data features of the corpus data to obtain target corpus data.
[0063] Among them, data features can refer to some features possessed by the corpus data itself, which can be obtained after analyzing multiple corpus data. In an embodiment of the present application, the information processing device can first filter the high-quality corpus data to be optimized according to the data features of the corpus data, and then further deduplicate the filtered corpus data to obtain the target corpus data. It should be noted that the target corpus data can be high-quality Chinese corpus data with coherent sentences and containing certain knowledge.
[0064] The information processing method provided in the embodiments of the present application can first screen the corpus data based on the target information, and then continue to screen the screened corpus data using the target quality classification model, and then further optimize the processing according to the data characteristics of the corpus data. That is, the corpus data can be screened multiple times using the target information, the target quality classification model and the data characteristics of the corpus data, thereby ensuring the quality of the target corpus data obtained, thereby solving the problem that the corpus cleaning solution in the related art cannot effectively solve the poor corpus quality, and greatly improving the quality of the corpus.
[0065] Based on the above embodiments, the present application provides an information processing method, referring to Figure 2 As shown, the method may include the following steps:
[0066] Step 201: The information processing device processes the corpus data to be processed based on the target data style to obtain processed corpus data.
[0067] The target data style may refer to a style of standardized corpus data. The information processing device may use the target data style of the standardized corpus data to process characters in the corpus data to be processed and obtain processed corpus data in a standardized format.
[0068] In other embodiments of the present application, step 201 may be implemented in the following manner:
[0069] Step 201a: The information processing device determines target characters in the characters of the corpus data to be processed based on the target data pattern.
[0070] Among them, the target data style can be compared and analyzed with the corpus data to be processed, and then the target characters that do not conform to the standardized target data style in the corpus data to be processed are extracted; that is, the target characters are the characters in the corpus data to be processed that do not conform to the standardized format of the high-quality corpus.
[0071] In a feasible implementation, the target character may refer to special characters, abnormal characters, redundant characters, traditional characters, etc.
[0072] Step 201b: The information processing device processes the target characters in the corpus data to be processed to obtain processed corpus data.
[0073] In an embodiment of the present application, the information processing device may delete and / or edit the target characters in the corpus to be processed to obtain processed corpus data. In a feasible implementation, the target characters of the types of special characters, abnormal characters, redundant characters, etc. in the corpus to be processed may be deleted, and the traditional characters in the corpus to be processed may be converted into simplified characters, thereby obtaining processed corpus data.
[0074] Step 202: The information processing device removes target information from the processed corpus data to obtain filtered corpus data.
[0075] Among them, information related to the target information can be first determined from the processed corpus data, and the determined information related to the target information can be directly deleted from the processed corpus data, thereby obtaining the filtered corpus data.
[0076] In other embodiments of the present application, step 202 may be implemented in the following manner:
[0077] Step 202a: The information processing device uses a target elimination technique to eliminate target information from the processed corpus data to obtain filtered corpus data.
[0078] The target screening technology may refer to a screening technology capable of deleting specific information; in a feasible implementation, the target screening technology may refer to a regular expression. That is, regular expressions and other technical means may be used to delete target information in the processed corpus data to obtain filtered corpus data.
[0079] In a feasible implementation, the filtered corpus data can be obtained by an information processing device using regular expressions to delete information such as Email, URL, IP address, ID number, instant messaging tool number, HTML tags, telephone number, corporate information, banned words, sensitive words, etc. from the processed corpus data.
[0080] Step 203: The information processing device determines high-quality sample corpus data and low-quality sample corpus data.
[0081] Among them, high-quality sample corpus data can refer to a large amount of corpus with relatively good quality and used for model training, and low-quality sample corpus data can refer to a large amount of corpus with relatively poor quality and used for model training.
[0082] In other embodiments of the present application, step 203 may be implemented in the following manner:
[0083] Step 203a: The information processing device obtains first sub-high-quality sample corpus data from the target high-quality corpus database.
[0084] In an embodiment of the present application, the target high-quality corpus database may refer to a database that includes all high-quality corpora. In a feasible implementation, the target high-quality corpus database may refer to a Chinese high-quality corpus database, and in particular may refer to the WuDaoCorpora dataset. Among them, the first sub-high-quality sample corpus data may be a certain amount of corpora directly extracted from the WuDaoCorpora dataset.
[0085] Step 203b: The information processing device determines sample corpus data to be classified.
[0086] The sample corpus data to be classified may refer to a large amount of sample corpus that has not been processed and classified and is initially obtained; it should be noted that the sample corpus data to be classified may include various types of corpus, such as high-quality corpus and low-quality corpus.
[0087] Step 203c: The information processing device screens and processes the sample corpus data to be classified to obtain low-quality sample corpus data and second sub-high-quality sample corpus data.
[0088] In the embodiments of the present application, Figure 3 As shown, the sample corpus data to be classified obtained by sampling can be screened through repeated data, keyword density matching and semantic clustering screening; then, the screened sample corpus data to be classified is manually assisted in labeling; finally, the high-quality corpus in the screened sample corpus data to be classified is extracted using the information of manual assisted labeling to obtain the second sub-high-quality sample corpus data, and the low-quality corpus in the screened sample corpus data to be classified is extracted to obtain the low-quality sample corpus data.
[0089] It should be noted that the labels of current low-quality corpus mainly include invalid content (such as advertisements, website navigation, comments, etc.), biased content (dirty words, sensitive words, etc.), and semantic incoherence (incomplete sentences, insufficient content expression, etc.).
[0090] Step 203d: The information processing device determines high-quality sample corpus data based on the first sub-high-quality sample corpus data and the second sub-high-quality sample corpus data.
[0091] The obtained first sub-high-quality sample corpus data and the obtained second sub-high-quality sample corpus data may be combined together to obtain high-quality sample corpus data.
[0092] Step 204: The information processing device performs binary classification model training on the target pre-trained language representation model based on the high-quality sample corpus data and the low-quality sample corpus data to obtain a target quality classification model.
[0093] Among them, the target quality classification model can divide the corpus data into high-quality corpus data and low-quality corpus data; the low-quality corpus data at least includes corpus data with invalid content, biased content and semantically incoherent content.
[0094] In the embodiments of the present application, Figure 3 As shown in the figure, the high-quality sample corpus data can be divided into high-quality training sample corpus data and high-quality test sample corpus data, and the low-quality corpus data can be divided into low-quality training sample corpus data and low-quality test sample corpus data; then, the high-quality training sample corpus data and low-quality training sample corpus data (i.e., quality classification training set) are used to perform supervised binary classification training and tuning on the target pre-trained language representation model, and a quality classification model to be verified that can identify three types of low-quality corpora, namely, invalid content, biased content, and semantic incoherence is trained; then, the high-quality test sample corpus data and low-quality test sample corpus data (i.e., quality classification test set) can be used to evaluate the quality classification model to be verified, and analyze whether there is misclassification of the low-quality corpus on the test set. If the corpus quality is high, it will be included in the high-quality training set. At the same time, analyze whether the high-quality corpus in the test set has been omitted or deleted, and add obviously unreasonable data to the low-quality corpus. Repeat this process multiple times, continuously expand the training set and optimize the quality classification model until the quality classification model achieves the expected effect, thereby obtaining the target quality classification model; it should be noted that the classification accuracy of the expected specific quality classification model in this embodiment reaches 90%. In a feasible implementation method, the target pre-trained language representation model may refer to a bidirectional encoder representation from a transformer (Bidirectional Encoder Representations from Transformers, BERT) model; the target quality classification model may refer to a BERT model after optimized training.
[0095] Step 205: The information processing device processes the screened corpus data using the target quality classification model to obtain high-quality corpus data to be optimized.
[0096] In an embodiment of the present application, the high-quality corpus data to be optimized may be the result of inputting the processed corpus data into the optimized and trained BERT model, and processing the processed corpus data through the optimized and trained BERT model and then outputting it.
[0097] Step 206: The information processing device filters the high-quality corpus data to be optimized based on the data features to obtain filtered corpus data.
[0098] In the embodiment of the present application, the information processing device may first set some screening conditions according to the data characteristics of the corpus data, and then filter the high-quality corpus data to be optimized according to the screening conditions, thereby obtaining the filtered corpus data. It should be noted that the quality of the filtered corpus data is higher than the quality of the high-quality corpus data to be optimized.
[0099] It should be noted that step 206 can be implemented in the following ways:
[0100] Step 206a: The information processing device determines a target filtering condition based on the data characteristics.
[0101] Among them, the target filtering condition can be set according to the data characteristics of the corpus data, and is used to optimize the filtering rules for the high-quality corpus data to be optimized; in a feasible implementation method, the target filtering condition can include at least one of the following: deleting sentences (articles) in which punctuation marks account for more than 50% of the overall length of the article, deleting useless sentences in the article based on regular / string fuzzy matching, such as seeking attention, seeking forwarding, etc., filtering short texts, deleting sentences starting with (login, registration, etc.), and the sentence length is less than 10, filtering and discarding website navigation, directory, title, comments, etc. based on html element information, deleting pure timestamps, etc.
[0102] Step 206b: The information processing device filters the high-quality corpus data to be optimized based on the target filtering condition to obtain filtered corpus data.
[0103] Among them, the sentences, articles, characters, etc. in the high-quality corpus data to be optimized can be filtered according to the filtering rules set in the target filtering conditions to obtain the filtered corpus data.
[0104] Step 207: The information processing device performs deduplication processing on the filtered corpus data to obtain target corpus data.
[0105] In an embodiment of the present application, repeated information in the filtered corpus data can be deleted to obtain the target corpus data. Specifically, the filtered corpus data can be subjected to deterministic deduplication, fuzzy deduplication and / or semantic similarity deduplication. Among them, deterministic deduplication can refer to deleting the content of the hit template based on common substring fuzzy matching; fuzzy deduplication can refer to removing articles with a duplication degree exceeding 0.9 based on minimum hash (MinHash) and SimHash; semantic similarity deduplication can refer to deleting semantically similar sentences in the same article based on the bert_score semantic similarity model.
[0106] It should be noted that step 207 can be implemented by at least one of the following steps:
[0107] Step 207a: The information processing device performs deduplication processing on the filtered corpus data based on the target substring to obtain target corpus data.
[0108] The target substring may refer to a common substring; the information processing device may perform fuzzy matching according to the template corresponding to the common substring to delete the content that hits the template in the filtered corpus data, thereby obtaining the target corpus data.
[0109] And / or, step 207b, the information processing device determines the similarity between the filtered corpus data, and performs deduplication processing on the filtered corpus data based on the similarity to obtain the target corpus data.
[0110] In an embodiment of the present application, the similarity between the filtered corpus data can be determined based on MinHash or SimHash; then, the articles whose similarity meets the target similarity threshold in the filtered corpus data can be removed to obtain the target corpus data. In a feasible implementation, the target similarity threshold can be a preset value, such as 0.9; that is, the articles whose similarity is greater than 0.9 in the filtered corpus data can be removed.
[0111] And / or, step 207c: the information processing device deletes sentences whose semantic similarity meets the target similarity in the filtered corpus data to obtain the target corpus data.
[0112] In an embodiment of the present application, the bert_score semantic similarity model can be used to calculate the semantic similarity between the filtered corpus data, and sentences whose semantic similarity in the filtered corpus data meets the target similarity are deleted to obtain the target corpus data. The target similarity can be a similarity threshold representing semantic similarity.
[0113] In other embodiments of the present application, Figure 4 As shown, after obtaining the original data of the corpus data to be processed, the corpus data to be processed can be subjected to format normalization, data desensitization, processing using the target quality analysis model, heuristic rule filtering and data deduplication in sequence to obtain the target corpus data. Among them, format normalization can be achieved through step 201, data desensitization can be achieved through step 202, processing using the target quality analysis model can be achieved through steps 203 to 205, heuristic rule filtering can be achieved through step 206, and data deduplication can be achieved through step 207.
[0114] Moreover, the information processing method in the present application can obtain high-quality Chinese corpus with coherent sentences and certain knowledge after corpus cleaning from low-quality general-domain corpus and security-domain corpus in Chinese. In this way, the target corpus data obtained can provide the model with more comprehensive and in-depth language knowledge, thereby improving its performance and effect in various natural language processing tasks; specifically, it can improve the performance of the model: the pre-training corpus can provide the model with rich background knowledge, helping the model to better understand the language meaning and contextual information, thereby improving the performance of the model in Chinese natural language processing tasks; it can also reduce the demand for data volume: using the target corpus data can reduce the demand for large-scale Chinese corpora, because the target corpus data already contains rich information, and the model can achieve good performance on a smaller data set; it can also improve the versatility of the model: the target corpus data can improve the versatility of the model, so that the model can achieve good performance in a variety of different Chinese natural language processing tasks, such as text classification, sentiment analysis, named entity recognition, etc.
[0115] It should be noted that, for the description of the same steps and the same contents in this embodiment as those in other embodiments, reference can be made to the description in other embodiments and will not be repeated here.
[0116] The information processing method provided in the embodiments of the present application can first screen the corpus data based on the target information, then continue to screen the screened corpus data using the target quality classification model, and then further filter according to the data features of the corpus data. That is, the corpus data can be screened multiple times using the target information, the target quality classification model and the data features of the corpus data to ensure the quality of the target corpus data obtained, thereby solving the problem that the corpus cleaning solution in the related art cannot effectively solve the poor corpus quality, and greatly improving the quality of the corpus.
[0117] Based on the above embodiments, the present invention provides an information processing device, which can be applied to Figure 1 and 2 In the information processing method provided in the corresponding embodiment, refer to Figure 5 As shown, the information processing device 3 may include: a screening unit 31, a processing unit 32 and an optimization unit 33, wherein:
[0118] A screening unit 31 is used to screen the corpus data to be processed based on the target information to obtain screened corpus data; wherein the target information at least includes information related to the user's privacy information and target words;
[0119] A processing unit 32 is used to process the screened corpus data using a target quality classification model to obtain high-quality corpus data to be optimized;
[0120] The optimization unit 33 is used to optimize the high-quality corpus data to be optimized based on the data features of the corpus data to obtain target corpus data.
[0121] In other embodiments of the present application, the screening unit 31 is further configured to perform the following steps:
[0122] Processing the corpus data to be processed based on the target data style to obtain processed corpus data;
[0123] The target information in the processed corpus data is eliminated to obtain the filtered corpus data.
[0124] In other embodiments of the present application, the screening unit 31 is further configured to perform the following steps:
[0125] Determining target characters among characters of the corpus data to be processed based on the target data pattern;
[0126] Processing target characters in the corpus data to be processed to obtain processed corpus data;
[0127] Correspondingly, the screening unit 31 is further used to remove target information from the processed corpus data by using a target screening technology to obtain screened corpus data.
[0128] In other embodiments of the present application, refer to Figure 5 As shown, the information processing device 3 may further include: a determination unit 34 and a training unit 35, wherein:
[0129] A determination unit 34, used to determine high-quality sample corpus data and low-quality sample corpus data;
[0130] A training unit 35 is used to perform binary classification model training on the target pre-trained language representation model based on the high-quality sample corpus data and the low-quality sample corpus data to obtain a target quality classification model;
[0131] Among them, the target quality classification model can divide the corpus data into high-quality corpus data and low-quality corpus data; the low-quality corpus data at least includes corpus data with invalid content, biased content and semantically incoherent content.
[0132] In other embodiments of the present application, the determining unit 34 is further configured to perform the following steps:
[0133] Acquire the first sub-high-quality sample corpus data from the target high-quality corpus database;
[0134] Determine the sample corpus data to be classified;
[0135] Screening and processing the sample corpus data to be classified to obtain low-quality sample corpus data and second-sub high-quality sample corpus data;
[0136] Based on the first sub-high-quality sample corpus data and the second sub-high-quality sample corpus data, high-quality sample corpus data is determined.
[0137] In other embodiments of the present application, the optimization unit 33 is further configured to perform the following steps:
[0138] Filter the high-quality corpus data to be optimized based on data features to obtain filtered corpus data;
[0139] The filtered corpus data is deduplicated to obtain the target corpus data.
[0140] In other embodiments of the present application, the optimization unit 33 is further configured to perform the following steps:
[0141] Determine target filtering conditions based on data characteristics;
[0142] The high-quality corpus data to be optimized is filtered based on the target filtering conditions to obtain filtered corpus data.
[0143] In other embodiments of the present application, the optimization unit 33 is further configured to perform the following steps:
[0144] Based on the target substring, the filtered corpus data is deduplicated to obtain the target corpus data;
[0145] and / or, determining the similarity between the filtered corpus data, and performing deduplication processing on the filtered corpus data based on the similarity to obtain target corpus data;
[0146] And / or, sentences whose semantic similarity meets the target similarity in the filtered corpus data are deleted to obtain the target corpus data.
[0147] It should be noted that the specific description of the steps performed by each unit can be found in Figure 1 and 2 The information processing method provided in the corresponding embodiment will not be described in detail here.
[0148] The information processing device provided in the embodiments of the present application can first screen the corpus data based on the target information, and then continue to screen the screened corpus data using the target quality classification model, and then further filter according to the data features of the corpus data, that is, the corpus data can be screened multiple times using the target information, the target quality classification model and the data features of the corpus data, thereby ensuring the quality of the target corpus data obtained, thereby solving the problem that the corpus cleaning solution in the related art cannot effectively solve the poor corpus quality, and greatly improving the quality of the corpus.
[0149] Based on the above embodiments, the embodiments of the present application provide an information processing device, which can be applied to Figure 1 and 2 In the information processing method provided in the corresponding embodiment, refer to Figure 6 As shown, the information processing device 4 may include: a processor 41, a memory 42 and a communication bus 43, wherein:
[0150] The communication bus 43 is used to realize the communication connection between the processor 41 and the memory 42;
[0151] The processor 41 is used to execute the information processing program in the memory 42 to implement the following steps:
[0152] The corpus data to be processed is screened based on the target information to obtain screened corpus data; wherein the target information at least includes information related to the user's private information and target words;
[0153] The target quality classification model is used to process the screened corpus data to obtain high-quality corpus data to be optimized;
[0154] The high-quality corpus data to be optimized is optimized based on the data features of the corpus data to obtain the target corpus data.
[0155] In other embodiments of the present application, the processor 41 is used to execute the information processing program in the memory 42 to filter the corpus data to be processed based on the target information to obtain the filtered corpus data, so as to implement the following steps:
[0156] Processing the corpus data to be processed based on the target data style to obtain processed corpus data;
[0157] The target information in the processed corpus data is eliminated to obtain the filtered corpus data.
[0158] In other embodiments of the present application, the processor 41 is used to execute the information processing program in the memory 42 to process the corpus data to be processed based on the target data style to obtain the processed corpus data, so as to implement the following steps:
[0159] Determining target characters among characters of the corpus data to be processed based on the target data pattern;
[0160] Processing target characters in the corpus data to be processed to obtain processed corpus data;
[0161] Correspondingly, the processor 41 is used to execute the information processing program in the memory 42 to remove the target information from the processed corpus data, and obtain the filtered corpus data, so as to implement the following steps:
[0162] The target elimination technology is used to eliminate the target information in the processed corpus data to obtain the filtered corpus data.
[0163] In other embodiments of the present application, the processor 41 is used to execute the information processing program in the memory 42, and can also implement the following steps:
[0164] Determine high-quality sample corpus data and low-quality sample corpus data;
[0165] Based on high-quality sample corpus data and low-quality sample corpus data, the target pre-trained language representation model is trained for binary classification to obtain a target quality classification model;
[0166] Among them, the target quality classification model can divide the corpus data into high-quality corpus data and low-quality corpus data; the low-quality corpus data at least includes corpus data with invalid content, biased content and semantically incoherent content.
[0167] In other embodiments of the present application, the processor 41 is used to execute the information processing program in the memory 42 to determine the high-quality sample corpus data and the low-quality sample corpus data to implement the following steps:
[0168] Acquire the first sub-high-quality sample corpus data from the target high-quality corpus database;
[0169] Determine the sample corpus data to be classified;
[0170] Screening and processing the sample corpus data to be classified to obtain low-quality sample corpus data and second-sub high-quality sample corpus data;
[0171] Based on the first sub-high-quality sample corpus data and the second sub-high-quality sample corpus data, high-quality sample corpus data is determined.
[0172] In other embodiments of the present application, the processor 41 is used to execute the information processing program in the memory 42 to optimize the high-quality corpus data to be optimized based on the data features of the corpus data to obtain the target corpus data, so as to implement the following steps:
[0173] Filter the high-quality corpus data to be optimized based on data features to obtain filtered corpus data;
[0174] The filtered corpus data is deduplicated to obtain the target corpus data.
[0175] In other embodiments of the present application, the processor 41 is used to execute the information processing program in the memory 42 to filter the high-quality corpus data to be optimized based on data features to obtain filtered corpus data, so as to implement the following steps:
[0176] Determine target filtering conditions based on data characteristics;
[0177] The high-quality corpus data to be optimized is filtered based on the target filtering conditions to obtain filtered corpus data.
[0178] In other embodiments of the present application, the processor 41 is used to execute the information processing program in the memory 42 to perform deduplication processing on the filtered corpus data to obtain the target corpus data, so as to implement the following steps:
[0179] Based on the target substring, the filtered corpus data is deduplicated to obtain the target corpus data;
[0180] and / or, determining the similarity between the filtered corpus data, and performing deduplication processing on the filtered corpus data based on the similarity to obtain target corpus data;
[0181] And / or, sentences whose semantic similarity meets the target similarity in the filtered corpus data are deleted to obtain the target corpus data.
[0182] It should be noted that the specific description of the steps executed by the processor can be referred to Figure 1 and 2 The information processing method provided in the corresponding embodiment will not be described in detail here.
[0183] The information processing device provided by the embodiments of the present application can first screen the corpus data based on the target information, and then continue to screen the screened corpus data using the target quality classification model, and then further filter according to the data features of the corpus data, that is, the corpus data can be screened multiple times using the target information, the target quality classification model and the data features of the corpus data, thereby ensuring the quality of the target corpus data obtained, thereby solving the problem that the corpus cleaning solution in the related art cannot effectively solve the poor corpus quality, and greatly improving the quality of the corpus.
[0184] Based on the foregoing embodiments, the embodiments of the present application provide a computer-readable storage medium, which stores one or more programs, and the one or more programs can be executed by one or more processors to implement Figure 1 and 2 The corresponding embodiments provide steps of the information processing method.
[0185] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of hardware embodiments, software embodiments, or embodiments in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage and optical storage, etc.) that contain computer-usable program code.
[0186] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data transmission device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data transmission device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0187] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data transmission device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0188] These computer program instructions may also be loaded onto a computer or other programmable data transmission device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0189] The above description is only a preferred embodiment of the present application and is not intended to limit the protection scope of the present application.
Claims
1. An information processing method, characterized in that: The method comprises: Based on the target information, the corpus data to be processed is screened to obtain screened corpus data; wherein the target information at least includes information related to the user's information and target words; Using a target quality classification model to process the screened corpus data to obtain high-quality corpus data to be optimized; The high-quality corpus data to be optimized is optimized based on the data features of the corpus data to obtain target corpus data.
2. The method according to claim 1, characterized in that The filtering and processing of the corpus data to be processed based on the target information to obtain the filtered corpus data includes: Processing the to-be-processed corpus data based on the target data style to obtain processed corpus data; The target information is removed from the processed corpus data to obtain the filtered corpus data.
3. The method according to claim 2, characterized in that The step of processing the to-be-processed corpus data based on the target data style to obtain the processed corpus data includes: Based on the target data pattern, determining target characters among the characters of the corpus data to be processed; Processing the target characters in the corpus data to be processed to obtain the processed corpus data; Correspondingly, removing the target information from the processed corpus data to obtain the filtered corpus data includes: The target information in the processed corpus data is removed by using a target screening technique to obtain the screened corpus data.
4. The method according to claim 1, characterized in that: The method further comprises: Determine high-quality sample corpus data and low-quality sample corpus data; Based on the high-quality sample corpus data and the low-quality sample corpus data, a binary classification model training is performed on the target pre-trained language representation model to obtain the target quality classification model; wherein the target quality classification model is capable of dividing the corpus data into high-quality corpus data and low-quality corpus data; the low-quality corpus data at least includes corpus data of invalid content, biased content, and semantically incoherent types.
5. The method according to claim 4, characterized in that The determining of high-quality sample corpus data and low-quality sample corpus data includes: Acquire the first sub-high-quality sample corpus data from the target high-quality corpus database; Determine the sample corpus data to be classified; Screening and processing the sample corpus data to be classified to obtain the low-quality sample corpus data and a second sub-high-quality sample corpus data; The high-quality sample corpus data is determined based on the first sub-high-quality sample corpus data and the second sub-high-quality sample corpus data.
6. The method according to claim 1, characterized in that The optimizing process of the high-quality corpus data to be optimized based on the data features of the corpus data to obtain target corpus data includes: Filtering the high-quality corpus data to be optimized based on the data features to obtain filtered corpus data; The filtered corpus data is deduplicated to obtain the target corpus data.
7. The method according to claim 6, characterized in that The filtering the high-quality corpus data to be optimized based on the data features to obtain filtered corpus data includes: Determining a target filtering condition based on the data characteristics; The high-quality corpus data to be optimized is filtered based on the target filtering condition to obtain the filtered corpus data.
8. The method according to claim 6, characterized in that The step of performing deduplication processing on the filtered corpus data to obtain the target corpus data includes: Performing deduplication processing on the filtered corpus data based on the target substring to obtain the target corpus data; and / or, determining the similarity between the filtered corpus data, and performing deduplication processing on the filtered corpus data based on the similarity to obtain the target corpus data; And / or, deleting sentences whose semantic similarity meets the target similarity in the filtered corpus data to obtain the target corpus data.
9. An information processing device, characterized in that: The device comprises: A screening unit, configured to screen the corpus data to be processed based on target information to obtain screened corpus data; wherein the target information at least includes information related to the user's private information and target words; A processing unit, used to process the screened corpus data using a target quality classification model to obtain high-quality corpus data to be optimized; The optimization unit is used to optimize the high-quality corpus data to be optimized based on the data features of the corpus data to obtain target corpus data.
10. An information processing device, characterized in that: The device comprises: a processor, a memory and a communication bus; The communication bus is used to realize the communication connection between the processor and the memory; The processor is used to execute the information processing program in the memory to implement the steps of the information processing method according to any one of claims 1 to 8.
11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps of the information processing method according to any one of claims 1 to 8.