Cleaning method, device and equipment for large model data set and medium

Abstract

The invention relates to the field of data cleaning, particularly provides a cleaning method and device for a large model data set, equipment and a medium, and aims to solve the problem of low data cleaning accuracy. In order to achieve the purpose, the method comprises the steps of obtaining a to-be-cleaned data set; performing classification processing on the to-be-cleaned data set according to a first pre-training model to obtain a positive sample data set and a negative sample data set; respectively carrying out clustering processing on the positive sample data set and the negative sample data set to obtain a positive sample clustering data set and a negative sample clustering data set; performing sampling processing on each classification data set according to a positive sample proportion corresponding to each classification data set in the positive sample clustering data set and the negative sample clustering data set to obtain a first data set; wherein the positive sample proportion corresponding to each classification data set is obtained by performing verification processing on each classification data set in the positive sample clustering data set and the negative sample clustering data set.

Description

Technical Field

[0001] The present application relates to the field of data cleaning, and specifically provides a method, device, equipment and medium for cleaning a large model data set. Background Art

[0002] In the field of data science and artificial intelligence, depending on the problem to be solved, it is necessary to collect data in the field corresponding to the problem and use the data for data analysis and model training. In order to ensure the accuracy of data analysis and models, it is generally necessary to clean the collected data before data analysis and model training. Data cleaning refers to the process of cleaning, organizing, converting and expanding the collected raw data to make it suitable for data analysis and model training.

[0003] In the prior art, it is usually necessary to collect data and annotate the collected data to obtain a labeled data set, and use the data set to train the constructed model to obtain a pre-trained model, and then use the pre-trained model to classify the data obtained by the crawler to obtain the target data set, thereby achieving data cleaning of the data obtained by the crawler. This method can achieve data cleaning to a certain extent, but after using the constructed model to clean the data obtained by the crawler, there is still a lot of illegal data in the target data set, which leads to the technical problem of low data cleaning accuracy. Summary of the invention

[0004] In order to overcome the above-mentioned defects, the present application is proposed to solve or at least partially solve the technical problem of low data cleaning accuracy in data cleaning.

[0005] In a first aspect, the present application provides a method for cleaning a large model data set, comprising:

[0006] Get the data set to be cleaned;

[0007] According to the first pre-training model, the to-be-cleaned data set is classified to obtain a positive sample data set and a negative sample data set;

[0008] Clustering the positive sample data set and the negative sample data set to obtain a positive sample cluster data set and a negative sample cluster data set;

[0009] According to the positive sample ratio corresponding to each classification data set in the positive sample clustering data set and the negative sample clustering data set, sampling processing is performed on each classification data set to obtain a first data set; wherein the positive sample ratio corresponding to each classification data set is obtained by verifying each classification data set in the positive sample clustering data set and the negative sample clustering data set.

[0010] In a technical solution of the above-mentioned method for cleaning a large model data set, the method further includes:

[0011] In the i-th round, classifying the i-th data set according to the first pre-training model to obtain the i-th positive sample data set and the i-th negative sample data set;

[0012] Performing clustering processing on the i-th positive sample data set and the i-th negative sample data set respectively to obtain an i-th positive sample cluster data set and an i-th negative sample cluster data set;

[0013] According to the i-th positive sample rate corresponding to each classified data set in the i-th positive sample cluster data set and the i-th negative sample cluster data set, sampling processing is performed on each classified data set to obtain an i+1-th data set; wherein the i-th positive sample rate corresponding to each classified data set is obtained by verifying each classified data set in the i-th positive sample cluster data set and the i-th negative sample cluster data set;

[0014] As shown in FIG. 1≤i≤I, where I corresponds to the average value of the Ith positive sample rate corresponding to each classification data set obtained after the Ith round is greater than the preset average value.

[0015] In a technical solution of the above-mentioned method for cleaning a large model data set, obtaining the first pre-trained model includes:

[0016] Get data sets in preset fields;

[0017] According to preset conditions, the data set in the preset field is screened to obtain a training data set;

[0018] According to the training data set, the preset model is trained to obtain the first pre-trained model.

[0019] In a technical solution of the above-mentioned method for cleaning a large model data set, the step of obtaining the data set to be cleaned includes:

[0020] Get the dataset;

[0021] According to preset rules, the data set is screened and deduplicated to obtain a preliminary screening data set;

[0022] The preliminary screening data set is classified and processed according to the second pre-training model to obtain the data set to be cleaned.

[0023] In a technical solution of the above-mentioned method for cleaning a large model data set, clustering the positive sample data set and the negative sample data set respectively to obtain a positive sample cluster data set and a negative sample cluster data set includes:

[0024] According to the preset number of categories, preset parameters and preset clustering model, clustering processing is performed on the positive sample data set and the negative sample data set respectively to obtain a positive sample cluster data set and a negative sample cluster data set.

[0025] In a technical solution of the above-mentioned large model data set cleaning method, the positive sample ratio corresponding to each classification data set is obtained by verifying each classification data set in the positive sample clustering data set and the negative sample clustering data set, including:

[0026] Extracting sample data from each classification data set in the first positive sample clustering data set and the first negative sample clustering data set;

[0027] Based on verifying the sample data corresponding to each classification data set, the positive sample ratio corresponding to each classification data set is obtained.

[0028] In a technical solution of the above-mentioned method for cleaning a large model data set, the source of the data set of the preset field includes:

[0029] Forums, news, video sites, books.

[0030] In a second aspect, the present application provides a data cleaning device, comprising:

[0031] An acquisition module is used to obtain the data set to be cleaned;

[0032] A classification module, used for classifying the data set to be cleaned according to the first pre-trained model to obtain a positive sample data set and a negative sample data set;

[0033] A clustering module, used to cluster the positive sample data set and the negative sample data set respectively to obtain a positive sample cluster data set and a negative sample cluster data set;

[0034] The sampling module is used to perform sampling processing on each classified data set in the positive sample cluster data set and the negative sample cluster data set according to the positive sample ratio corresponding to each classified data set to obtain a first data set.

[0035] In a third aspect, the present application provides an electronic device, comprising at least one processor and a memory; wherein:

[0036] The memory stores computer-executable instructions;

[0037] The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor performs the method as described in any one of the first aspects.

[0038] In a fourth aspect, the present application provides a computer-readable storage medium having a plurality of program codes stored therein, wherein the program codes are suitable for being loaded and executed by a processor to execute the method as described in any one of the first aspects.

[0039] The present application provides a method, device, equipment and medium for cleaning a data set of a large model. The method specifically comprises: obtaining a data set to be cleaned; classifying the data set to be cleaned according to a first pre-trained model to obtain a positive sample data set and a negative sample data set; clustering the positive sample data set and the negative sample data set respectively to obtain a positive sample cluster data set and a negative sample cluster data set; sampling each classified data set according to the positive sample cluster data set and the negative sample cluster data set corresponding to the positive sample cluster data set to obtain a first data set; wherein the positive sample ratio corresponding to each classified data set is obtained by verifying each classified data set in the positive sample cluster data set and the negative sample cluster data set, so that after the data set is cleaned, a very small number of negative samples are contained in the cleaned data set, thereby improving the accuracy of data cleaning. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] The disclosure of the present application will become more easily understood with reference to the accompanying drawings. It is easy for those skilled in the art to understand that these drawings are only for illustrative purposes and are not intended to limit the scope of protection of the present application. In addition, similar numbers in the drawings are used to represent similar components, among which:

[0041] Figure 1 A schematic diagram of a process flow of a data cleaning method embodiment 1 provided in an embodiment of the present application;

[0042] Figure 2 A flowchart of a second embodiment of a data cleaning method provided in an embodiment of the present application;

[0043] Figure 3 A flowchart of a third embodiment of a data cleaning method provided in an embodiment of the present application;

[0044] Figure 4 A flowchart of a fourth embodiment of a data cleaning method provided in an embodiment of the present application;

[0045] Figure 5 A flowchart of a fifth embodiment of a data cleaning method provided in an embodiment of the present application;

[0046] Figure 6 A flowchart of a sixth embodiment of a data cleaning method provided in an embodiment of the present application;

[0047] Figure 7A schematic diagram of the structure of a data cleaning device embodiment 1 provided in an embodiment of the present application;

[0048] Figure 8 A schematic diagram of the structure of an electronic device embodiment 1 provided in an embodiment of the present application.

[0049] Reference numerals list :

[0050] 11: acquisition module; 12: classification module; 13: clustering module; 14: sampling module; 21: processor; 22: memory. DETAILED DESCRIPTION

[0051] Some embodiments of the present application are described below with reference to the accompanying drawings. Those skilled in the art should understand that these embodiments are only used to explain the technical principles of the present application and are not intended to limit the protection scope of the present application.

[0052] In the description of the present application, "module" and "processor" may include hardware, software or a combination of the two. A module may include hardware circuits, various suitable sensors, communication ports, memory, and may also include software parts, such as program code, or a combination of software and hardware. The processor may be a central processing unit, a microprocessor, an image processor, a digital signal processor or any other suitable processor. The processor has data and / or signal processing functions. The processor may be implemented in software, hardware or a combination of the two. Non-temporary computer-readable storage media include any suitable medium that can store program code, such as a disk, a hard disk, an optical disk, a flash memory, a read-only memory, a random access memory, etc. The term "A and / or B" means all possible combinations of A and B, such as only A, only B or A and B. The term "at least one A or B" or "at least one of A and B" has a similar meaning to "A and / or B" and may include only A, only B or A and B. The singular terms "one" and "the" may also include plural forms.

[0053] At present, the traditional data cleaning methods mainly include the following two methods:

[0054] First, a seed dataset of the target domain is collected. Then, positive samples are extracted according to the defined domain of the seed dataset, and corresponding negative samples are collected from the web crawler dataset. Then, a model is trained to classify the data into two categories. The web crawler dataset is classified using the model to obtain the confidence of each data in the dataset, and the confidence of each data is sorted. According to the sorting result, the data selection confidence threshold is set. ChatGPT (ChatGenerator Pre-trained Transformer) is used to verify the domain relevance and reduce false positives. Unconfirmed positive samples are added to the training set and the above process is repeated.

[0055] TOP2Vec (Topic to Vector) first creates a joint embedding of document and word vectors; after the document and word vectors are embedded in the vector space, the dimensionality of the data is reduced; the algorithm performs a clustering operation on the data after dimensionality reduction, grouping similar documents together to form different topics; finally, the algorithm sorts the word vectors according to their importance in their respective topics, which can help users understand the topic content more easily and find key information.

[0056] BERTopic (Bidirectional Encoder Representations from Transformers Topic) uses embedding techniques to extract document embeddings; reduce the dimensionality of the embeddings; cluster the reduced embeddings; extract and refine topics to increase the consistency of the most marginally associated words, thereby creating a representation of the topic.

[0057] The above data cleaning methods still contain a lot of dirty data after data cleaning, which leads to the technical problem of low data cleaning accuracy.

[0058] Based on this, in order to solve the above technical problems, the technical concept of this application is: how to provide a new data cleaning method to improve the accuracy of data cleaning.

[0059] Figure 1 The following is a flow chart of a data cleaning method embodiment 1 provided in the present application. Figure 1 As shown, specifically, the method includes:

[0060] Step S101: Obtain a data set to be cleaned.

[0061] In this embodiment, the data set to be cleaned can be obtained from the Internet, from an enterprise database, or from a public data set by crawling.

[0062] In this embodiment, for example, a semi-crawler approach may be used to collect data in the field of logistics insurance from online forums, legal cases, news, contracts, Wikipedia, textbooks, and video websites.

[0063] Step S102: Classify the data set to be cleaned according to the first pre-training model to obtain a positive sample data set and a negative sample data set.

[0064] In this embodiment, the data in the data set to be cleaned are input into the first pre-training model one by one to obtain the confidence that each data belongs to a positive sample, and the data with a confidence greater than or equal to 0.6 is used as a positive sample, and the data with a confidence less than 0.6 is used as a negative sample. The positive samples are stored in the positive sample data set, and the negative samples are stored in the negative sample data set.

[0065] In this embodiment, the positive sample data set and the negative sample data set may be stored in a hard disk, or in a database or in the cloud.

[0066] In this embodiment, after the data set to be cleaned is classified and processed by the first pre-trained model, since the first pre-trained model obtains the probability that a certain data in the cleaned data set belongs to a positive sample, rather than completely confirming that the data set is a positive sample, both the positive sample data set and the negative sample data set may contain positive samples and negative samples.

[0067] Step S103: clustering the positive sample data set and the negative sample data set respectively to obtain a positive sample cluster data set and a negative sample cluster data set.

[0068] In this embodiment, after clustering the positive sample data set, multiple data sets of different categories are obtained, which are called positive sample cluster data sets. After clustering the negative sample data set, multiple data sets of different categories are obtained, which are called negative sample cluster data sets.

[0069] In this embodiment, the positive sample data set and the negative sample data set are clustered and divided into data sets of multiple categories, so as to facilitate subsequent data cleaning of the data sets of multiple categories.

[0070] Step S104: performing sampling processing on each classified data set according to the positive sample ratio corresponding to each classified data set in the positive sample cluster data set and the negative sample cluster data set to obtain a first data set.

[0071] In this embodiment, the positive sample ratio corresponding to each classification data set is obtained by verifying each classification data set in the positive sample clustering data set and the negative sample clustering data set.

[0072] In this embodiment, the verification can be performed manually or the data can be automatically verified according to preset rules.

[0073] In this embodiment, according to the positive sample ratio corresponding to each classification data set, the positive sample ratio is multiplied by the number of classification data sets corresponding to the positive sample ratio to obtain the amount of data sampled from the classification data set.

[0074] In this embodiment, for one of the classification data sets, a portion of the data in the classification data set is obtained for verification, and after the verification processing of the portion of the data, the number of positive samples in the classification data set is obtained, and the ratio of the number of positive samples to the number of the portion of data is the positive sample ratio.

[0075] In this embodiment, a data set to be cleaned is obtained; according to the first pre-trained model, the data set to be cleaned is classified and processed to obtain a positive sample data set and a negative sample data set; the positive sample data set and the negative sample data set are clustered and processed respectively to obtain a positive sample clustered data set and a negative sample clustered data set; according to the positive sample ratio corresponding to each classified data set in the positive sample clustered data set and the negative sample clustered data set, each classified data set is sampled and processed to obtain a first data set. Compared with the prior art in which data is cleaned only according to the pre-trained model, the data after data cleaning still contains a large amount of dirty data, which leads to a technical problem of low data cleaning accuracy, the present application classifies the data to be cleaned by using the first pre-trained model to obtain a positive sample data set and a negative sample data set, and clusters the positive sample data set and the negative sample data set respectively to obtain a positive sample clustered data set and a negative sample clustered data set, and then according to the positive sample ratio corresponding to each classified data set in the positive sample clustered data set and the negative sample clustered data set, each classified data set is sampled and processed to obtain the first data set, thereby improving the accuracy of the cleaned data.

[0076] Figure 2 A flow chart of a second embodiment of a data cleaning method provided in an embodiment of the present application is provided. Based on the above embodiment, Figure 2 As shown, specifically, the method further includes:

[0077] Step S201: In the i-th round, the i-th data set is classified according to the first pre-training model to obtain the i-th positive sample data set and the i-th negative sample data set.

[0078] In this embodiment, the data set needs to be cleaned for multiple rounds. In the i-th round, each data in the i-th data set is classified according to the first pre-trained model to obtain the confidence that each data belongs to a positive sample, and the data with a confidence greater than or equal to a preset confidence threshold is stored in the i-th positive sample data set, and the data with a confidence less than the preset confidence threshold is stored in the i-th negative sample data set.

[0079] In this embodiment, 1≤i≤I, where I is the total number of rounds after data cleaning is completed.

[0080] Step S202: clustering the i-th positive sample data set and the i-th negative sample data set respectively to obtain the i-th positive sample cluster data set and the i-th negative sample cluster data set.

[0081] In this embodiment, the i-th positive sample clustering data set and the i-th negative sample clustering data set both contain multiple classification data sets. The data in the same classification data set have commonalities. Therefore, after clustering, multiple classification data sets are obtained, and each classification data set is different.

[0082] Step S203: According to the i-th positive sample rate corresponding to each classified data set in the i-th positive sample cluster data set and the i-th negative sample cluster data set, sampling processing is performed on each classified data set to obtain the i+1-th data set.

[0083] In this embodiment, the i-th positive sample rate corresponding to each classified data set is obtained by verifying each classified data set in the i-th positive sample cluster data set and the i-th negative sample cluster data set.

[0084] In this embodiment, for one of the classified data sets, since all the data therein have something in common, a part of the data in one of the classified data sets is verified, and the positive sample rate in the part of the data set obtained can be used as the i-th positive sample rate of the classified data.

[0085] Step S204: Obtain the average value of the i-th positive sample rate corresponding to each classification data set. If it is determined that the average value is less than the preset average value, execute steps S201 to S203; otherwise, end.

[0086] In this embodiment, if the average value of the i-th positive sample rate corresponding to each classification data set is greater than the preset average value, it is considered that the number of positive samples in the i+1-th data set accounts for a high proportion, and data cleaning can be terminated at this time.

[0087] In this embodiment, in the i-th round, according to the first pre-training model, the i-th data set is classified to obtain the i-th positive sample data set and the i-th negative sample data set; the i-th positive sample data set and the i-th negative sample data set are clustered to obtain the i-th positive sample cluster data set and the i-th negative sample cluster data set; according to the i-th positive sample rate corresponding to each classified data set in the i-th positive sample cluster data set and the i-th negative sample cluster data set, each classified data set is sampled to obtain the i+1-th data set; the average value of the i-th positive sample rate corresponding to each classified data set is obtained, and if it is determined that the average value is less than the preset average value, steps S201 to S203 are executed; otherwise, the process ends, so as to achieve multiple rounds of data cleaning, thereby improving the accuracy of data cleaning.

[0088] Figure 3 The flowchart of the third embodiment of a data cleaning method provided in the present application is based on the above embodiment. Figure 3 As shown, obtaining the first pre-trained model in step S102 includes:

[0089] Step S301: Acquire a data set of a preset domain.

[0090] In this embodiment, for example, a semi-crawler method is used to collect nearly 10 million pieces of data in the field of logistics insurance from forums, news, video websites, and books. The data types of these data include text data, image data containing text, and video data containing subtitles.

[0091] Step S302: According to preset conditions, the data set in the preset field is screened to obtain a training data set.

[0092] In this embodiment, the data set of the preset field is screened according to the keywords.

[0093] In this embodiment, for example, keywords are set in the GPT-4o (Generated Pre-trained Transformer-4omni, fourth generation universal generative pre-trained transformer) model, and data is input into the GPT-4o model to obtain a training data set.

[0094] In this embodiment, since the data set in the preset field contains a small amount of negative samples, after the data set in the preset field is screened using keywords, the training data set obtained contains a very small amount of negative samples.

[0095] Step S303: According to the training data set, the preset model is trained to obtain a first pre-trained model.

[0096] In this embodiment, the training data set is input into a preset model for training to obtain a first pre-trained model.

[0097] In this embodiment, for example, the preset model can be a FastText (Fast Text) model, a VIT (Visual Transformer) model, or a CLIP (Contrast Language-Image Pre-training) model.

[0098] In this embodiment, a data set of a preset field is obtained; according to preset conditions, the data set of the preset field is screened and processed to obtain a training data set; according to the training data set, a preset model is trained and processed to obtain a first pre-trained model, so as to obtain a first pre-trained model with high accuracy without data labeling.

[0099] Figure 4 A flowchart of a fourth embodiment of a data cleaning method provided in an embodiment of the present application is provided. Based on the above embodiment, Figure 4 As shown, an implementation of step S101 specifically includes:

[0100] Step S401: Acquire a data set.

[0101] In this embodiment, the data set may be obtained from the Internet, from an enterprise database, or from a public data set by crawling.

[0102] In this embodiment, for example, the dataset may be CC (Conceptual Captions), CCI (Chinese Internet Corpus), or RefinedWeb (The RefinedWeb Dataset).

[0103] Step S402: According to preset rules, the data set is screened and deduplicated to obtain a preliminary screening data set.

[0104] In this embodiment, the common points of the same data are used as preset rules to deduplicate the duplicate data in the data set to obtain a deduplicated data set, and the deduplicated data set is screened according to keywords in a preset field to obtain a preliminary screening data set.

[0105] Step S403: classify the initial screening data set according to the second pre-training model to obtain the data set to be cleaned.

[0106] In this embodiment, the second pre-training model can divide the data in the primary screening data set into different categories according to the subject, and use the data belonging to the preset field as positive samples and other data as negative samples. The collection of positive samples and negative samples is used as the data set to be cleaned.

[0107] In this embodiment, for example, the BERTopic model can be used to classify the primary screening data set according to its type and title, and the data belonging to the preset field is used as a positive sample, and the other data is used as a negative sample. The collection of positive samples and negative samples is used as the data set to be cleaned.

[0108] In this embodiment, a data set is obtained; according to preset rules, the data set is screened and deduplicated to obtain a preliminary screening data set; the preliminary screening data set is classified according to the second pre-trained model to obtain a data set to be cleaned.

[0109] Figure 5 A flowchart of a fifth embodiment of a data cleaning method provided in an embodiment of the present application is provided. Based on the above embodiment, Figure 5 As shown, an implementation of step S103 specifically includes:

[0110] Step S501: According to the preset number of categories, preset parameters and preset clustering model, the positive sample data set and the negative sample data set are clustered to obtain a positive sample cluster data set and a negative sample cluster data set.

[0111] In this embodiment, for any one of the positive sample data set and the negative sample data set, according to the preset clustering model, firstly, samples of a preset number of categories are randomly selected from the data set as initial cluster centers, and then the distance between each sample in the data set and each cluster center is calculated, and each sample is assigned to the cluster center closest to it. For each cluster set, the mean of all samples within it is recalculated, and the mean is used as the new cluster center, and the above steps are repeated until the cluster center no longer changes.

[0112] In this embodiment, for example, the preset clustering model may be a k-means clustering algorithm.

[0113] In this embodiment, according to the preset number of categories, preset parameters and preset clustering model, the positive sample data set and the negative sample data set are clustered respectively to obtain the positive sample cluster data set and the negative sample cluster data set, so as to classify the positive sample cluster data set and the negative sample cluster data set to achieve more refined classification of the data.

[0114] Figure 6 A flowchart of a sixth embodiment of a data cleaning method provided in an embodiment of the present application is provided. Based on the above embodiment, Figure 6 As shown, in step S104, the positive sample ratio corresponding to each classification data set in the positive sample clustering data set and the negative sample clustering data set is obtained, which specifically includes:

[0115] Step S601: extracting sample data from each classification data set in the first positive sample clustering data set and the first negative sample clustering data set.

[0116] In this embodiment, a portion of data is extracted from each classification data set in the first positive sample cluster data set and the first negative sample cluster data set as sample data.

[0117] In this embodiment, for example, according to uniform distribution, a portion of data is randomly extracted from each classification data set in the first positive sample clustering data set and the first negative sample clustering data set as sample data.

[0118] Step S602: Based on the verification processing of the sample data corresponding to each classification data set, the positive sample ratio corresponding to each classification data set is obtained.

[0119] In this embodiment, the sample data may be manually verified or automatically verified according to preset rules to obtain the positive sample ratio corresponding to each classification data set.

[0120] In this embodiment, sample data is extracted from each classification data set in the first positive sample cluster data set and the first negative sample cluster data set; based on verification processing of the sample data corresponding to each classification data set, the positive sample ratio corresponding to each classification data set is obtained.

[0121] It should be pointed out that although the various steps in the above embodiments are described in a specific order, those skilled in the art can understand that in order to achieve the effect of the present application, different steps do not have to be executed in such an order. They can be executed simultaneously (in parallel) or in other orders. These changes are within the scope of protection of the present invention.

[0122] Furthermore, the present application also provides a data cleaning device.

[0123] Figure 7 This is a structural diagram of a data cleaning device embodiment 1 provided in the present application. Figure 7As shown, the data cleaning device in the embodiment of the present application mainly includes one or more of the acquisition module 11, the classification module 12, the clustering module 13 and the sampling module 14, which can be combined into one module. In some embodiments, the acquisition module 11 can be configured to acquire the data set to be cleaned. The classification module 12 can be configured to classify the data set to be cleaned according to the first pre-trained model to obtain a positive sample data set and a negative sample data set. The clustering module 13 can be configured to cluster the positive sample data set and the negative sample data set respectively to obtain a positive sample clustering data set and a negative sample clustering data set. The sampling module 14 can be configured to sample each classified data set according to the positive sample clustering data set and the negative sample clustering data set The corresponding positive sample ratio of each classified data set, to obtain a first data set.

[0124] The data cleaning device is used to perform Figures 1 to 6 The data cleaning method embodiments shown in the figure are similar in technical principles, technical problems solved and technical effects produced. Those skilled in the art can clearly understand that for the convenience and brevity of description, the specific working process and related instructions of the data cleaning device can refer to the contents described in the embodiment of the data cleaning method in which the execution subject is the data cleaning device, and will not be repeated here.

[0125] It is understood by those skilled in the art that all or part of the processes in the method for implementing the above-mentioned embodiment of the present application can also be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor, the steps of each of the above-mentioned method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable storage medium may include: any entity or device, medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory, random access memory, electric carrier signal, telecommunication signal and software distribution medium that can carry the computer program code. It should be noted that the content contained in the computer-readable storage medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable storage media do not include electric carrier signals and telecommunication signals.

[0126] Furthermore, the present application also provides an electronic device.

[0127] Figure 8 This is a schematic diagram of the structure of an electronic device embodiment 1 provided in the present application. Figure 7As shown, the electronic device includes at least one processor 21 and a memory 22, and the memory 22 can be configured to store and execute the above Figures 1 to 7 The program of the data cleaning method of the embodiment shown, the processor 21 can be configured to execute the program in the first memory, the program including but not limited to a program for executing a data cleaning method of the above method embodiment. For ease of explanation, only the part related to the embodiment of the present application is shown. For specific technical details not disclosed, please refer to the method part of the embodiment of the present application. The electronic device can be a control device device formed by various electronic devices.

[0128] Furthermore, the present application also provides a computer-readable storage medium. In a computer-readable storage medium embodiment according to the present application, the computer-readable storage medium may be configured to store the execution method Figures 1 to 6 The program of the data cleaning method of the illustrated embodiment can be loaded and run by a processor to implement the above-mentioned data cleaning method. For ease of explanation, only the parts related to the embodiments of the present application are shown. For specific technical details not disclosed, please refer to the method part of the embodiment of the present application. The computer-readable storage medium can be a storage device formed by various electronic devices. Optionally, the computer-readable storage medium in the embodiment of the present application is a non-temporary computer-readable storage medium.

[0129] Further, it should be understood that since the setting of each module is only for illustrating the functional units of the device of the present application, the physical devices corresponding to these modules may be the processor itself, or a part of the software in the processor, a part of the hardware, or a part of the combination of software and hardware. Therefore, the number of each module in the figure is only schematic.

[0130] It is understood by those skilled in the art that each module in the device can be adaptively split or merged. Such splitting or merging of specific modules will not cause the technical solution to deviate from the principle of the present application, and therefore, the technical solutions after splitting or merging will fall within the protection scope of the present application.

[0131] So far, the technical solutions of the present application have been described in conjunction with the preferred embodiments shown in the accompanying drawings. However, it is easy for those skilled in the art to understand that the protection scope of the present application is obviously not limited to these specific embodiments. Without departing from the principles of the present application, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will fall within the protection scope of the present application.

Claims

1. A method for cleaning a large model data set, characterized in that: include: Get the data set to be cleaned; According to the first pre-training model, the to-be-cleaned data set is classified to obtain a positive sample data set and a negative sample data set; Clustering the positive sample data set and the negative sample data set to obtain a positive sample cluster data set and a negative sample cluster data set; According to the positive sample ratio corresponding to each classification data set in the positive sample clustering data set and the negative sample clustering data set, sampling processing is performed on each classification data set to obtain a first data set; The positive sample ratio corresponding to each classification data set is obtained by verifying each classification data set in the positive sample clustering data set and the negative sample clustering data set.

2. The method according to claim 1, characterized in that The method further comprises: In the i-th round, classifying the i-th data set according to the first pre-training model to obtain the i-th positive sample data set and the i-th negative sample data set; Performing clustering processing on the i-th positive sample data set and the i-th negative sample data set respectively to obtain an i-th positive sample cluster data set and an i-th negative sample cluster data set; According to the i-th positive sample rate corresponding to each classified data set in the i-th positive sample cluster data set and the i-th negative sample cluster data set, sampling processing is performed on each classified data set to obtain an i+1-th data set; wherein the i-th positive sample rate corresponding to each classified data set is obtained by verifying each classified data set in the i-th positive sample cluster data set and the i-th negative sample cluster data set; As shown in FIG. 1≤i≤I, where I corresponds to the average value of the Ith positive sample rate corresponding to each classification data set obtained after the Ith round is greater than the preset average value.

3. The method according to claim 1, characterized in that Acquiring the first pre-trained model includes: Get data sets in preset fields; According to preset conditions, the data set in the preset field is screened to obtain a training data set; According to the training data set, the preset model is trained to obtain the first pre-trained model.

4. The method according to claim 1, characterized in that The step of obtaining the data set to be cleaned includes: Get the dataset; According to preset rules, the data set is screened and deduplicated to obtain a preliminary screening data set; The preliminary screening data set is classified and processed according to the second pre-training model to obtain the data set to be cleaned.

5. The method according to claim 1, characterized in that The clustering of the positive sample data set and the negative sample data set to obtain a positive sample cluster data set and a negative sample cluster data set comprises: According to the preset number of categories, preset parameters and preset clustering model, clustering processing is performed on the positive sample data set and the negative sample data set respectively to obtain a positive sample cluster data set and a negative sample cluster data set.

6. The method according to claim 1, characterized in that The positive sample ratio corresponding to each classification data set is obtained by verifying each classification data set in the positive sample clustering data set and the negative sample clustering data set, including: Extracting sample data from each classification data set in the first positive sample clustering data set and the first negative sample clustering data set; Based on verifying the sample data corresponding to each classification data set, the positive sample ratio corresponding to each classification data set is obtained.

7. The method according to claim 3, characterized in that The sources of the datasets in the preset fields include: Forums, news, video sites, books.

8. A data cleaning device, characterized in that: include: An acquisition module is used to obtain the data set to be cleaned; A classification module, used for classifying the to-be-cleaned data set according to the first pre-trained model to obtain a positive sample data set and a negative sample data set; A clustering module, used for clustering the positive sample data set and the negative sample data set respectively to obtain a positive sample cluster data set and a negative sample cluster data set; The sampling module is used to perform sampling processing on each classified data set in the positive sample cluster data set and the negative sample cluster data set according to the positive sample ratio corresponding to each classified data set to obtain a first data set.

9. An electronic device, characterized in that: comprising at least one processor and a memory; wherein, The memory stores computer-executable instructions; The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor performs the method of any one of claims 1 to 7.

10. A computer-readable storage medium storing a plurality of program codes, characterized in that: The program code is suitable for being loaded and executed by a processor to execute the method according to any one of claims 1 to 7.

Images