A data cleaning method and related apparatus

Abstract

The application discloses a data cleaning method applied to a data cleaning process in the field of artificial intelligence (AI) technology. In the data cleaning method, for a data set to be cleaned, the type of a task to be executed after the data set is cleaned is determined, and a cleaning operation of the data set is determined according to the type of the task. Based on a specific cleaning mode corresponding to the cleaning operation, the data set can be analyzed to determine a cleaning condition of the data set based on an analysis result of the data set, and the data set is cleaned. In this way, the cleaning condition of the data is defined based on the characteristics of the task to which the data set is applied and the analysis result of the data in the data set, the data can be cleaned based on the most fundamental data cleaning purpose and the characteristics of the data itself, the accuracy of the data cleaning is improved, the cleaning of the data can be completed at one time, and the data cleaning efficiency is improved.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a data cleaning method and related apparatus. Background Technology

[0002] In the field of artificial intelligence (AI) technology, large models refer to machine learning models with complex computational structures and a massive number of parameters. Large models are capable of processing massive amounts of data and performing various complex tasks, such as natural language processing and image recognition.

[0003] Currently, the effectiveness of training large-scale models is highly dependent on the quality of the data. High-quality, large-scale, and diverse datasets can effectively improve the accuracy of large-scale models and reduce their training time. Because large-scale models require high-quality datasets, it is often necessary to clean the pre-collected data before training them, thereby removing low-quality data to ensure that the input data meets the requirements of the large-scale model.

[0004] In related technologies, data cleaning metrics are typically specified manually based on experience, and data cleaning is then performed based on these manually specified metrics, such as cleaning text data with fewer than N words. However, manually specifying data cleaning metrics may result in data that does not meet the required quality, leading to poor model training performance. Therefore, when manually specifying cleaning metrics, users often need to repeatedly adjust the cleaning metrics and perform multiple cleaning operations on the dataset to obtain data of satisfactory quality, resulting in low data cleaning efficiency. Summary of the Invention

[0005] This application provides a data cleaning method that can improve the efficiency of data cleaning.

[0006] Firstly, a data cleaning method is provided, applied to the data cleaning process in the field of AI technology. This data cleaning method includes: an execution device first obtaining the task type of the task to be processed in the dataset to be cleaned. The task to be processed in the dataset can be understood as the task to be performed after the dataset is cleaned. That is, based on the aforementioned task type, the execution device can determine what type of task the cleaned dataset will be used for.

[0007] Then, the execution device determines the cleaning operations to be performed on the dataset based on the task type. That is, the task type determines which cleaning operations need to be used when cleaning the dataset. Different task types often correspond to different cleaning operations, ensuring that the cleaned dataset meets the requirements of the task type.

[0008] Secondly, the execution device analyzes the dataset according to the cleaning method corresponding to the cleaning operation, and obtains the analysis results. The cleaning method indicates how the data in the dataset should be cleaned (e.g., based on which information in the data is used for cleaning). Therefore, based on the determined cleaning method, the execution device can analyze the data in the dataset to obtain the analysis results, thereby acquiring specific information about the data in the dataset related to the cleaning method.

[0009] After obtaining the analysis results, the execution device retrieves the corresponding cleaning conditions for the cleaning operation based on those results. In other words, the cleaning conditions essentially indicate the data cleaning standards, specifying which conditions the data must meet to require cleaning.

[0010] Finally, the execution device performs cleaning operations on the data in the dataset according to the cleaning conditions. That is, based on the specific conditions indicated by the cleaning indicators, the execution device can determine which data in the dataset meets the cleaning requirements, and then remove the data that meets the cleaning requirements, thereby achieving data cleaning of the dataset.

[0011] In this solution, for a dataset requiring cleaning, the type of task to be performed after cleaning is first determined, and the cleaning operation is then determined based on the task type. Based on the specific cleaning method corresponding to the cleaning operation, the dataset can be analyzed to determine the cleaning conditions, thereby achieving dataset cleaning. In this way, defining the data cleaning conditions based on the characteristics of the task applied to the dataset and the analysis results of the data allows for targeted data cleaning based on the fundamental data cleaning objectives and the characteristics of the data itself, improving the accuracy of data cleaning, ensuring that data cleaning can be completed in one go, and thus improving data cleaning efficiency.

[0012] In one possible implementation, when determining the cleaning operation for data cleaning based on the task type, the execution device may specifically input the task type into a first artificial intelligence inference model, which then outputs the cleaning operation. The first artificial intelligence inference model is a pre-trained model used to predict the cleaning operation to be used during data cleaning. In another possible implementation, when obtaining the cleaning conditions corresponding to the cleaning operation based on the analysis results, the execution device may specifically input the analysis results into a second artificial intelligence inference model, which then outputs the cleaning conditions. The second artificial intelligence inference model is a pre-trained model used to predict the cleaning conditions to be used during data cleaning based on the analysis results of the dataset.

[0013] In one possible implementation, before obtaining the task type of the dataset to be cleaned, the execution device performs preset basic cleaning operations on the dataset. These basic cleaning operations are applicable to datasets corresponding to different task types. In other words, for datasets used to perform different types of tasks, the aforementioned basic cleaning operations can be used to perform data cleaning and complete the basic cleaning of the dataset.

[0014] In one possible implementation, when analyzing a dataset according to the cleaning method corresponding to the cleaning operation, the execution device specifically determines the attribute values ​​of the data in the dataset under the target attribute based on the cleaning method corresponding to the cleaning operation, obtaining a set of attribute values. The analysis result includes the set of attribute values. The aforementioned cleaning method includes performing data cleaning based on the attribute values ​​of the data under the target attribute.

[0015] In other words, the process of performing analysis on a dataset is actually based on the dataset cleaning method to statistically analyze the specific attribute values ​​of the data in the dataset under the target attribute, thereby obtaining the attribute values ​​of each data in the dataset.

[0016] In one possible implementation, the cleaning conditions include a first condition, which indicates that the value of a first attribute of the data to be cleaned is within a first range, and the target attribute mentioned above includes the first attribute.

[0017] Therefore, when performing cleaning operations on the data in the dataset according to the cleaning conditions, the execution device can remove data in the dataset whose first attribute value is within a first range based on the set of attribute values.

[0018] In one possible implementation, when obtaining the cleaning conditions corresponding to the cleaning operation based on the analysis results, the execution device first obtains the data retention conditions, which are used to indicate the conditions that the amount of data retained after the dataset is cleaned must meet.

[0019] Furthermore, based on the data retention conditions and analysis results, the device obtains the corresponding cleaning conditions for the cleaning operation, so that the data retained after cleaning the dataset based on the cleaning conditions can meet the data retention conditions.

[0020] In this solution, by setting data retention conditions and combining these conditions with the attribute value set of the dataset to determine data cleaning metrics, it is possible to ensure that when the dataset is cleaned based on the determined cleaning conditions, the amount of data retained after cleaning can meet the requirements of the data retention conditions. This ensures that the amount of data retained after cleaning meets the needs of subsequent tasks and improves the accuracy of data cleaning.

[0021] In one possible implementation, the task type mentioned above is used to indicate the type of model training task, and the data in the dataset is text data, image data, video data, or audio data.

[0022] Secondly, a data cleaning apparatus is provided, comprising: an acquisition module for acquiring the task type of the task to be cleaned in the dataset; a processing module for determining the cleaning operation to be performed on the dataset according to the task type; the processing module for analyzing the dataset according to the cleaning method corresponding to the cleaning operation and obtaining the analysis result; the acquisition module for acquiring the cleaning conditions corresponding to the cleaning operation according to the analysis result; and the processing module for performing the cleaning operation on the data in the dataset according to the cleaning conditions.

[0023] In one possible implementation, when the processing module determines the cleaning operation for cleaning the dataset based on the task type, it specifically performs the following: inputs the task type into the first artificial intelligence inference model, and the first artificial intelligence inference model outputs the cleaning operation.

[0024] In one possible implementation, when the acquisition module obtains the cleaning conditions corresponding to the cleaning operation based on the analysis results, it is specifically used to: input the analysis results into the second artificial intelligence inference model, and have the second artificial intelligence model output the cleaning conditions.

[0025] In one possible implementation, before the acquisition module obtains the task type of the task to be cleaned, the processing module is also used to perform preset basic cleaning operations on the dataset.

[0026] In one possible implementation, when the processing module analyzes the dataset according to the cleaning method corresponding to the cleaning operation, it specifically performs the following: determines the attribute values ​​of the data in the dataset under the target attribute according to the cleaning method corresponding to the cleaning operation, obtains the attribute value set, and the analysis result includes the attribute value set; wherein, the cleaning method includes performing data cleaning based on the attribute values ​​of the data under the target attribute.

[0027] In one possible implementation, the cleaning conditions include a first condition, which indicates that the value of the first attribute of the data to be cleaned is within a first range, and the target attribute includes the first attribute; when the processing module performs a cleaning operation on the data in the dataset according to the cleaning conditions, it is specifically used to: based on the set of attribute values, remove data in the dataset whose value of the first attribute is within the first range.

[0028] In one possible implementation, when the acquisition module obtains the cleaning conditions corresponding to the request cleaning operation based on the analysis results, it is specifically used to: obtain data retention conditions, which are used to indicate the conditions that the amount of data retained after the dataset is cleaned must meet; and obtain the cleaning conditions corresponding to the request cleaning operation based on the data retention conditions and the analysis results.

[0029] In one possible implementation, the task type is used to indicate the type of model training task, where the data in the dataset is text data, image data, video data, or audio data.

[0030] Thirdly, a data cleaning apparatus is provided, comprising: a processor and a memory; the memory is used to store computer instructions, which, when executed by the processor, cause the data cleaning apparatus to perform the methods described above.

[0031] Fourthly, a computer-readable storage medium is provided that stores instructions which, when executed on a computer, cause the computer to perform the methods of any of the above aspects.

[0032] Fifthly, a computer program product containing instructions is provided, which, when executed on a computer, enable the computer to perform the methods described above.

[0033] In a sixth aspect, a chip system is provided, the chip system including a processor and a communication interface for communicating with a module other than the chip shown, the processor for running computer programs or instructions such that an apparatus on which the chip system is mounted can perform the methods of any of the above aspects.

[0034] In a seventh aspect, a computing device is provided, which includes a data cleaning apparatus of the third aspect or a chip system of the sixth aspect, wherein the data cleaning apparatus or chip system in the computing device is used to implement the operational steps of the method of any of the above aspects.

[0035] Eighthly, a computing device cluster is provided, comprising at least one computing device, wherein any one computing device is used to run a computer program or instructions, such that the computing device cluster can perform the methods of any of the above aspects. Alternatively, some or all of the computing devices are used together to run a computer program or instructions, such that the computing device cluster can perform the methods of any of the above aspects.

[0036] Based on the implementation methods provided in the above aspects, this application can be further combined to provide more implementation methods. Attached Figure Description

[0037] Figure 1 A schematic diagram of a system architecture provided for this application;

[0038] Figure 2 A flowchart illustrating a data cleaning method provided in this application;

[0039] Figure 3 A flowchart illustrating a process for determining data cleaning metrics based on a second artificial intelligence inference model, provided for this application;

[0040] Figure 4 A schematic diagram of a system architecture provided for this application;

[0041] Figure 5 A schematic diagram of a data cleaning process provided for this application;

[0042] Figure 6 A schematic diagram illustrating the data cleaning process executed by a module in a system architecture provided in this application;

[0043] Figure 7 A classification diagram of some common operators provided for this application;

[0044] Figure 8 A comparative illustration of text before and after marking, provided for this application;

[0045] Figure 9 A schematic diagram of an operator fusion optimization provided in this application;

[0046] Figure 10 A schematic diagram of the structure of a data cleaning device provided in this application;

[0047] Figure 11 A schematic diagram of the structure of a computing device provided in this application;

[0048] Figure 12 This application provides a schematic diagram of the structure of a computing device cluster;

[0049] Figure 13 This application provides a schematic diagram of another computing device cluster structure.

[0050] Figure 14 This is a schematic diagram of the structure of a computer-readable storage medium provided in this application. Detailed Implementation

[0051] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application are described below with reference to the accompanying drawings. Obviously, the described embodiments are merely some, and not all, of the embodiments of this application. Those skilled in the art will recognize that, with the emergence of new application scenarios, the technical solutions provided by this application are also applicable to similar technical problems.

[0052] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such descriptions can be used interchangeably where appropriate to allow embodiments to be implemented in a sequence other than that illustrated or described in this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or device that includes a series of steps or modules is not necessarily limited to those explicitly listed, but may include other steps or modules not explicitly listed or inherent to such processes, methods, products, or devices. The naming or numbering of steps appearing in this application does not imply that the steps in the method flow must be performed in the chronological / logical order indicated by the naming or numbering. The execution order of named or numbered process steps can be changed according to the desired technical purpose, as long as the same or similar technical effect is achieved. The division of units in this application is a logical division. In practical applications, there may be other division methods. For example, multiple units may be combined or integrated into another system, or some features may be ignored or not executed. In addition, the shown or discussed mutual coupling, direct coupling, or communication connection may be through some interface, and the indirect coupling or communication connection between units may be electrical or other similar forms, none of which are limited in this application. Furthermore, the units or sub-units described as separate components may or may not be physically separated, may or may not be physical units, or may be distributed among multiple circuit units. Some or all of the units can be selected to achieve the purpose of the solution in this application according to actual needs.

[0053] Since the quality of training data significantly impacts model training effectiveness, related technologies often require data cleaning before training to obtain high-quality training data. Currently, data cleaning typically involves manual specification of cleaning metrics based on experience, and the cleaning process is then performed according to these manually defined metrics.

[0054] For example, when training a text processing model, the user manually specifies that text data with fewer than N words should be removed to eliminate text data with too little content. Similarly, when training an image processing model, the user manually specifies that image data with a resolution below a certain value should be removed to eliminate overly blurry images.

[0055] However, manually specifying data cleaning metrics may result in data of insufficient quality, leading to poor model training performance. For example, overly strict metrics may result in insufficient data for model training. Conversely, overly lenient metrics may result in too much low-quality data, negatively impacting model training accuracy.

[0056] Therefore, when cleaning metrics are manually specified, users often need to repeatedly adjust the cleaning metrics to perform multiple cleaning operations on the dataset in order to obtain data that meets the required quality, resulting in low data cleaning efficiency.

[0057] In view of this, this application provides a data cleaning method. For a dataset requiring cleaning, the method first determines the type of task the dataset will be used for after cleaning, and then determines the cleaning operation based on the task type. Based on the specific cleaning method corresponding to the cleaning operation, the dataset can be analyzed to determine the cleaning conditions, thereby achieving dataset cleaning. In this way, defining the data cleaning conditions based on the characteristics of the task applied to the dataset and the analysis results of the data in the dataset enables targeted data cleaning based on the fundamental data cleaning purpose and the characteristics of the data itself, improving the accuracy of data cleaning, ensuring that data cleaning can be completed in one go, and thus improving data cleaning efficiency.

[0058] Please see Figure 1 , Figure 1 This is a schematic diagram of a system architecture provided for this application. (For example...) Figure 1 As shown, in this system architecture, the execution device 10 for performing the data cleaning method provided in this application can be implemented by a physical host (computing device) or multiple physical hosts (computing device cluster).

[0059] In addition, the system architecture includes a data storage system 11, which stores data that needs to be cleaned, such as multimedia data like text, audio, image, or video data. Specifically, the data stored in the data storage system 11 can be applied to model training, model inference, or data query scenarios.

[0060] Optionally, for persistent data storage, the data storage system 11 can be located outside the execution device 10, exchanging data with the execution device 10 via a network. Alternatively, if the execution device 10 is a physical host, the data storage system 11 can also be located inside the execution device 10, such as exchanging data with the processor via a bus. In this case, the data storage system 11 functions as a hard disk. With the data storage system 11, the execution device 10 can clean the data stored in the data storage system 11 and store the cleaned data in the data storage system 11.

[0061] Optionally, the execution device 10 may be connected to a data usage device 12. This data usage device 12 is used to obtain cleaned data from the execution device 10 or the data storage system 11, and to perform data processing tasks based on the cleaned data, such as performing model training tasks based on the cleaned data. The data usage device 12 may be implemented by a single physical host (computing device) or multiple physical hosts (computing device cluster).

[0062] Please see Figure 2 , Figure 2 This is a flowchart illustrating a data cleaning method provided in this application. Figure 2 As shown, the data cleaning method provided in this application includes the following steps 201-205.

[0063] Step 201: Obtain the task type for processing the dataset to be cleaned.

[0064] In this application, the execution device can obtain the dataset from a storage device (such as a storage server or hard drive), or it can obtain the dataset to be cleaned by receiving data uploaded by the user. The dataset includes multiple data sets, and the data in the dataset needs to be cleaned to remove data that does not meet the quality requirements.

[0065] Optionally, the data in the dataset can be multimedia data such as text data, image data, video data, or audio data. Furthermore, the data in the dataset can be of the same type, such as all text data or all image data. The data in the dataset can also be of different types, such as including both text data and image data.

[0066] In addition to the dataset, the execution device can also obtain the task type of the task processing the dataset to be cleaned. The task processing the dataset can be understood as the task to be performed after the dataset is cleaned. That is, based on the task type mentioned above, the execution device can know what type of task the cleaned dataset will be used for. Furthermore, the task type can be, for example, user-specified, indicating the type of task configured to be performed after the dataset is cleaned.

[0067] Optionally, the cleaned dataset is specifically used for model training tasks; that is, the cleaned dataset is used as training data to perform subsequent model training tasks. Furthermore, the task type mentioned above specifically indicates the type of model training task. Generally, the purpose of a model training task is to train a model capable of performing a specific inference task. Therefore, the type of model training task can be represented by the type of task performed by the trained model. For example, the type of model training task could be: training a long text classification model, training a short text classification model, training a question-answering model in a financial scenario, training a code completion model, training an image classification model, training an image detection model, or training a speech recognition model, etc.

[0068] In general, the type of model training task can depend on the type of task the trained model will perform. Since different models can perform a wide variety of tasks, this application does not limit the specific type of model training task.

[0069] Step 202: Determine the cleaning operation when cleaning the dataset based on the task type.

[0070] In other words, the task type determines which cleaning operations are required when cleaning the dataset. Different task types often correspond to different cleaning operations, ensuring that the cleaned dataset meets the requirements of the task type.

[0071] For example, when the task type is specifically training a long text processing model, the cleaning operations determined by the task type may include: performing cleaning based on the number of words in the text data, or performing cleaning based on the average number of words per line in the text data. Similarly, when the task type is specifically training a sentiment analysis model, the cleaning operations determined by the task type may include: performing cleaning based on the category of the text data, or performing cleaning based on the diversity index of the text data.

[0072] Step 203: Analyze the dataset according to the cleaning method corresponding to the cleaning operation to obtain the analysis results.

[0073] The cleaning method corresponding to the cleaning operation specifies how the data in the dataset should be cleaned (e.g., based on which information in the data to perform cleaning). Therefore, based on the determined cleaning method, the execution device can analyze the data in the dataset to obtain the analysis results and acquire specific information about the data in the dataset related to the cleaning method.

[0074] When analyzing a dataset according to the cleaning method corresponding to the cleaning operation, the execution device specifically determines the attribute values ​​of the data in the dataset under the target attribute based on the cleaning method corresponding to the cleaning operation, obtaining a set of attribute values. The analysis result includes the set of attribute values. The aforementioned cleaning method includes performing data cleaning based on the attribute values ​​of the data under the target attribute.

[0075] In other words, the process of performing analysis on a dataset is actually based on the dataset cleaning method to statistically analyze the specific attribute values ​​of the data in the dataset under the target attribute, thereby obtaining the attribute values ​​of each data in the dataset.

[0076] In this application, the target attribute can be one or more attributes of the data. Therefore, when the execution device determines the attribute value of the data in the dataset under the target attribute, it is actually determining the attribute value of the data under the specified one or more attributes. The attributes of the data are related to the type of the data itself. Different types of data often have different attributes. For example, for text data, the attributes of text data may include, for example, the number of words in the text data, the average number of words per line (i.e., the average number of words per line in the entire text data), the number of lines of text, the category of text data (e.g., legal text, scientific text, news text, or academic paper text), the editing time of text data, and the size of text data. For image data, the attributes of image data may include, for example, the resolution of image data, the size of image data, the contrast of image data, the color of image data (black and white or color), and the category of image data (e.g., animal images, landscape images, human images, etc.).

[0077] Given a target attribute for the data, the execution device can statistically obtain the attribute value of each data point in the dataset under the target attribute, thus generating a set of attribute values. This set of attribute values ​​includes the attribute values ​​of multiple data points in the dataset under the target attribute. The attribute value of data under the target attribute refers to the specific numerical value of the data within that dimension. For different data sets, the attribute value of the same data under the same attribute may be the same or different.

[0078] Optionally, since different types of tasks may focus on different data attributes, the execution device can determine the target attribute based on the task type. That is, the target attribute is related to the type of task performed after dataset cleaning. For example, if the dataset is cleaned and used to train a long text classification model, then the long text classification model focuses more on attributes such as the word count and category of the text data; or, if the dataset is cleaned and used to train a question-answering model in a financial scenario, then the long text classification model focuses more on attributes such as the average word count per line and the editing time of the text data.

[0079] In this way, by determining the attribute information that needs to be statistically analyzed in the dataset based on the type of task performed after data cleaning, it is possible to effectively identify which attribute information is truly relevant to subsequent tasks. Furthermore, based on the attribute information obtained from data statistics, higher-quality data can be selected for subsequent tasks, thereby improving the accuracy of data cleaning.

[0080] There are several ways to determine the target attributes based on the task type.

[0081] For example, a mapping relationship between task types and data attributes can be constructed manually based on expert experience or knowledge (such as research in existing papers), and the generated mapping relationship can be stored in a rule base. Then, by querying the mapping relationship in the rule base, the target attribute corresponding to the task type can be determined.

[0082] For example, when determining the cleaning operation for a dataset based on the task type, the execution device can specifically input the task type into a first artificial intelligence inference model, which then outputs the cleaning operation. In this way, after determining the cleaning method based on the cleaning operation, the aforementioned target attributes can be determined. Here, the first artificial intelligence inference model is a pre-trained model, and it is used to predict the cleaning operation used during data cleaning.

[0083] Specifically, the first AI inference model is, for example, an attribute prediction model. The execution device trains an attribute prediction model based on historical data related to data cleaning (such as the type of task applied after data cleaning, the attributes used during data cleaning, and the performance of the model trained after data cleaning). This attribute prediction model is then used to predict the attributes of interest during data cleaning based on the type of task performed after data cleaning. Thus, by inputting the target task type into the attribute prediction model, the target attribute output by the model can be obtained.

[0084] Step 204: Obtain the cleaning conditions corresponding to the washing operation based on the analysis results.

[0085] In other words, the cleaning conditions actually indicate the cleaning criteria for the data, specifying what conditions the data must meet to be cleaned.

[0086] Since the attribute value set includes attribute values ​​from multiple data points in the dataset, it effectively indicates the distribution of attribute values ​​for the target attribute across the dataset. Therefore, the execution device determines data cleaning metrics based on the distribution of attribute values ​​in the dataset and the type of task to be performed after dataset cleaning. These data cleaning metrics indicate the range of attribute values ​​in the data that needs cleaning (i.e., the data to be cleaned). In other words, data cleaning metrics essentially indicate the data cleaning standards, specifying the range of attribute values ​​within which cleaning is required.

[0087] Since data cleaning metrics are actually based on the specific attribute values ​​of the data, the setting of these metrics affects the degree of data cleaning. If the data cleaning metrics are set too strictly, less data may remain after cleaning, resulting in poor model training performance. Conversely, if the data cleaning metrics are set too leniently, some low-quality data may remain after cleaning, also leading to poor model training performance.

[0088] Furthermore, the requirements for data attribute values ​​in a dataset often differ depending on the task type. For example, if the cleaned dataset is used to train a long text classification model, then the training data for this model typically requires texts of considerable length; long text data is considered high-quality, while short text data is considered low-quality. Conversely, if the cleaned dataset is used to train a short text classification model, then the training data for the long text classification model typically requires texts of relatively short length; short text data is considered high-quality, while long text data is considered low-quality. In other words, different task types often have different standards for data quality. Therefore, the type of task performed after dataset cleansing is actually a crucial factor in determining the quality of the data.

[0089] Based on this, this step combines the type of task to be performed after dataset cleaning with the distribution of attribute values ​​of multiple data in the dataset to determine data cleaning indicators. This can effectively ensure that the cleaned data meets the requirements of the subsequent tasks, thereby improving the accuracy of data cleaning and making it easier to obtain high-quality data.

[0090] Step 205: Perform cleaning operations on the data in the dataset according to the cleaning conditions.

[0091] That is, the execution device can determine which data in the dataset meets the cleaning requirements based on the specific conditions indicated by the cleaning indicators, and then remove the data that meets the cleaning requirements to achieve data cleaning of the dataset.

[0092] Specifically, cleaning conditions can be a way of performing data cleaning based on the attribute values ​​of the data. Therefore, based on the attribute values ​​of multiple data in the attribute value set under the target attribute and the specific range indicated by the data cleaning indicators, it is possible to determine which data in the dataset meets the cleaning requirements, and then remove the data that meets the cleaning requirements to achieve data cleaning of the dataset and obtain the cleaned dataset.

[0093] For example, suppose the cleaning conditions include a first condition, which indicates that the value of a first attribute of the data to be cleaned is within a first range, and the target attribute includes the first attribute. Then, when performing a cleaning operation on the data in the dataset according to the cleaning conditions, the execution device can remove data in the dataset whose first attribute value is within the first range based on the set of attribute values.

[0094] For example, assuming the dataset contains text data, the first condition indicates whether the number of words in the text data to be cleaned is less than 20 or greater than 200. Therefore, if a piece of text data in the dataset has fewer than 20 words or more than 200 words, this text data can be removed during the data cleaning process.

[0095] Of course, besides the first condition, data cleaning conditions can also include other conditions. That is, data cleaning indicators can include multiple conditions, and different conditions can correspond to different attribute values. Therefore, when multiple cleaning conditions are involved, data cleaning can be performed on the dataset by removing data that meets any one of the cleaning conditions, thus obtaining the cleaned dataset. In other words, the attribute values ​​of the data included in the cleaned dataset are all outside the range indicated by the cleaning conditions.

[0096] Specifically, this solution defines the data cleaning method based on the characteristics of the task applied to the dataset and the attribute features of the data in the dataset. It can achieve targeted data cleaning based on the most fundamental data cleaning purpose and the characteristics of the data itself, improve the accuracy of data cleaning, ensure that data cleaning can be completed in one go, and thus improve the efficiency of data cleaning.

[0097] Optionally, in the process of determining data cleaning metrics as described in step 203 above, the execution device can acquire data retention conditions. These conditions indicate the quantity of data retained after dataset cleaning must meet certain criteria. For example, assuming the dataset contains M data points, the data retention condition could indicate that the number of retained data points after cleaning is not less than N, or that the number of retained data points is not less than N and not more than S (where N < S < M), or that the number of retained data points is not less than 30% or 50% of M, etc. In general, the data retention condition can specifically indicate the range within which the number of retained data points after dataset cleaning falls.

[0098] Then, the execution device can determine the cleaning conditions based on the data retention conditions and the analysis results (such as the set of attribute values ​​mentioned above), so that the data retained after cleaning the dataset based on the cleaning conditions can meet the data retention conditions.

[0099] In other words, when determining the cleaning conditions, the execution device also needs to consider the data retention conditions. This ensures that when cleaning the dataset based on the determined cleaning conditions, the amount of data retained after cleaning meets the requirements of the data retention conditions.

[0100] Understandably, for most model training tasks, in addition to the quality of the training data, the quantity of training data is often crucial. Insufficient training data often affects the accuracy and generalization ability of the trained model. Therefore, when using cleaned datasets for model training, it is necessary not only to ensure the quality of the cleaned dataset but also to ensure that the cleaned dataset contains sufficient data to train a model that meets the requirements.

[0101] In this solution, by setting data retention conditions and combining these conditions with the analysis results of the dataset to determine the cleaning conditions, it is ensured that when the dataset is cleaned based on the determined cleaning conditions, the amount of data retained after cleaning can meet the requirements of the data retention conditions. This ensures that the amount of data retained after cleaning meets the needs of subsequent tasks and improves the accuracy of data cleaning.

[0102] Optionally, data retention conditions are related to the task type and the categories of data in the dataset. In other words, data retention conditions can be specifically determined based on the task type and the categories of data in the dataset.

[0103] Specifically, in most model training tasks, the model needs to be trained using data from multiple different categories. Furthermore, to obtain a better model, the proportions of different categories in the training data may need to be maintained within a specific range. For example, in training a natural language model, the training data might include 30% code data, 30% mathematical data, and 40% text data. Moreover, different categories of data are often assigned to different datasets for separate data cleaning processes, and different datasets may undergo different data cleaning procedures.

[0104] Therefore, in the data cleaning process, in order to ensure that the data included in the final cleaned dataset can meet the requirements of subsequent tasks, the execution device can determine how much data needs to be retained after the dataset is cleaned based on the task type corresponding to the dataset and the data category in the dataset, thereby obtaining the data retention conditions.

[0105] In this solution, the data retention conditions are determined based on the task type corresponding to the dataset and the category of data in the dataset. This ensures that the retained data obtained after cleaning based on the data retention conditions meets the quantity requirements of subsequent tasks, ensures that the data distribution of subsequent tasks is within a reasonable range, and improves the accuracy of data cleaning.

[0106] Furthermore, in most scenarios, the set of attribute values ​​determined by the execution device includes a large number of attribute values ​​corresponding to different types of attributes. Therefore, the cleaning conditions determined based on this set of attribute values ​​often include conditions corresponding to different types of attributes, and these different conditions indicate the range of values ​​for different attributes corresponding to the data to be cleaned. Thus, when the cleaning conditions include a large number of conditions, and each condition can indicate the range of values ​​for a certain attribute, the process of determining the cleaning conditions is essentially selecting one combination of ranges from a large number of combinations, so that the final cleaning conditions can effectively complete the data cleaning process. Therefore, determining the corresponding conditions for different types of attribute values ​​to combine and obtain the cleaning conditions is actually a complex optimization process. To ensure that the execution device obtains accurate cleaning conditions, this application may employ a neural network model to determine the data cleaning indicators.

[0107] For example, the execution device can input the analysis results (such as the attribute value set mentioned above) and the task type into the second artificial intelligence inference model to obtain the cleaning conditions output by the second artificial intelligence inference model. The second artificial intelligence inference model is a pre-trained model used to predict the cleaning conditions to be used during data cleaning.

[0108] Of course, if the cleaning conditions also need to be determined based on the data retention conditions mentioned above, the execution device can input the data retention conditions, attribute value set and task type into the second artificial intelligence inference model to obtain the data cleaning indicators output by the second artificial intelligence inference model.

[0109] Specifically, the second AI inference model is a pre-trained neural network model used to predict cleaning conditions based on input data. During the training process of the second AI inference model, it can be trained based on historical data related to data cleaning over a historical period (such as data retention conditions, attribute values ​​of data in the dataset, the type of task applied after data cleaning, and the performance of the model trained after data cleaning) so that the second AI inference model can predict data cleaning indicators.

[0110] For example, please refer to Figure 3 , Figure 3 This application provides a flowchart illustrating a process for determining data cleaning metrics based on a second artificial intelligence inference model. For example... Figure 3 As shown, the execution device inputs data retention conditions, task type, and attribute value set into the target model to obtain the cleaning conditions output by the second artificial intelligence inference model. Specifically, the data retention condition indicates that after cleaning the dataset based on the cleaning conditions, 30% of the data in the dataset should be retained. The task type is a long text classification task, meaning the current dataset is used to train a long text classification model. The attribute value set includes the attribute values ​​of each text data in the dataset under various attributes, such as the word count of text 1-text N and the average word count per line of text 1-text N. Finally, the cleaning conditions output by the second artificial intelligence inference model can include multiple conditions, where condition 1 indicates a word count less than 20 or greater than 200, and condition 2 indicates an average word count per line of less than 4.

[0111] In addition to neural network models, the execution device may also use other machine learning-based prediction methods, such as linear regression or decision tree methods, to determine the cleaning conditions. This application does not specifically limit the method used by the execution device to determine the cleaning conditions.

[0112] The above describes the process of determining the target attribute based on the task type corresponding to the dataset, and then performing data cleaning based on the attribute values ​​of the data in the dataset under the target attribute and the task type. However, in some scenarios, data in datasets used to perform different types of tasks often need to undergo some of the same data cleaning processes. For example, when the data in the dataset is text data, the data cleaning process needs to remove text data edited in a non-specific language, that is, only retain text data in a specific language. Or, the data cleaning process needs to remove text data in the dataset containing sensitive content (such as violent or politically sensitive content). As another example, when the data in the dataset is image data, the data cleaning process needs to remove some particularly blurry image data, that is, only retain image data whose clarity meets specific requirements.

[0113] Based on this, in this application, the execution device can perform a basic cleaning process on the original dataset to obtain the aforementioned dataset. Then, the execution device performs the steps described in the above embodiments to perform a further data cleaning process on the dataset.

[0114] For example, before obtaining the task type of the dataset to be cleaned, the execution device first performs a preset basic cleaning operation on the dataset. This basic cleaning operation is applicable to datasets corresponding to different task types. In other words, the aforementioned basic cleaning operation can be used to perform data cleaning on datasets used to perform different types of tasks, thereby completing the basic cleaning of the dataset.

[0115] The aforementioned basic cleaning operations are used to instruct one or more operations to be performed on the data. The specific operations can be set according to the actual application scenario, and this application does not impose any specific limitations on them. For example, if the original dataset includes text data, the basic cleaning operations could be to remove text data edited in a non-specific language, text data containing sensitive content, or text data containing invalid content (such as garbled text). As another example, if the original dataset includes image data, the basic cleaning operations could be to remove image data whose clarity does not meet the requirements, etc.

[0116] Furthermore, in the actual data cleaning process, the execution device cleans the data in the original dataset according to the basic cleaning operations. Specifically, it can process the data in the original dataset by executing pre-set basic operators, thereby cleaning out data that does not meet the quality requirements.

[0117] The execution flow of a data cleaning method provided in this application has been described above. For ease of understanding, the execution process of this data cleaning method in practical applications will be detailed below with specific examples.

[0118] In practical applications, the data cleaning method provided in this application can be applied to an execution device, which can be a single server or a distributed data processing system composed of multiple servers. The following will use the application of the data cleaning method to a distributed data processing system as an example to describe the execution process of the data cleaning method.

[0119] Please see Figure 4 , Figure 4 This is a schematic diagram of a system architecture provided for this application. (For example...) Figure 4 As shown, the system architecture used in the data cleaning method includes a compute layer and a storage layer. The compute layer includes a master node and one or more slave nodes, while the storage layer includes one or more storage devices. Furthermore, the compute layer and the storage layer are connected via a network or cable.

[0120] In the computation layer, both master and slave nodes can be composed of one or more servers. Alternatively, master and slave nodes can be virtual machines deployed on servers. The master node includes a PipelineGenerator, a Task RuleManager, a Metric RuleManager, and a Pipeline Manager. Optionally, the master node may also include a Pipeline Executor. Slave nodes include Pipeline Executors. The PipelineGenerator is used to generate data cleaning-related operators. The Task RuleManager determines the required metric operators based on the type of task to be performed on the cleaned dataset, allowing for the statistical analysis of attribute values ​​for each data point. The Metric RuleManager selects hyperparameters for the cleaning operators based on the statistical information obtained from the metric operators. The Pipeline Manager manages cluster resource allocation and schedules the Pipeline Executor to execute operators. The pipeline execution module is used to execute various operators in the data cleaning pipeline to complete the data cleaning process.

[0121] In addition, the storage layer includes one or more storage devices for storing the dataset before and after cleaning. The master and slave nodes of the computation layer can read the dataset before cleaning from the storage devices in the storage layer, and store the dataset after cleaning into the storage devices.

[0122] For example, please refer to Figure 5 , Figure 5 A schematic diagram of a data cleaning process provided for this application; Figure 6 This is a schematic diagram illustrating the data cleaning process executed by a module in a system architecture provided in this application. Figure 5 As shown, the data cleaning process can be divided into three stages, each involving the generation and execution of a specific type of operator to complete the overall data cleaning process. Figure 5 and Figure 6 As shown, the data cleaning process includes the following steps 1-10, where stage one includes steps 1-3, stage two includes steps 4-6, and stage three includes steps 7-10.

[0123] Step 1: The pipeline generation module in the master node obtains the dataset to be cleaned and the type of downstream task to be executed after the dataset is cleaned.

[0124] During the data cleaning process, users can input the dataset to be cleaned and the type of downstream task to be executed after data cleaning into the pipeline generation module. Since the dataset to be cleaned often contains a large amount of data and is usually pre-stored on a storage device in the storage layer, users can specify the storage path of the dataset on the storage device to the pipeline generation module, enabling the module to read the dataset based on the specified storage path.

[0125] Step 2: The pipeline generation module generates basic operators.

[0126] Specifically, the pipeline generation module can generate basic operators based on the type of data in the dataset to be cleaned (such as text data, image data, or video data). These basic operators are independent of the downstream tasks performed after dataset cleaning; they are operators that need to be executed throughout the data cleaning process for the same type of data. For example, basic operators include data reading operators, data writing operators, personal privacy masking operators, sensitive content recognition operators, and language recognition operators.

[0127] For example, please refer to Figure 7 , Figure 7 A classification diagram of some common operators provided for this application. For example... Figure 7As shown, common operators can generally be classified into read / write operators, normalizers, classifiers, and scorers. Read / write operators are responsible for reading and writing data; for example, they can include JSON read operators, image read operators, and JSON write operators, which are responsible for reading and writing data in different formats.

[0128] Transformation operators are responsible for converting text data into a certain format. For example, the SpaceNormalizer converts various types of spaces into English spaces; the End PunctuationNormalizer removes paragraphs without ending punctuation (symbols that can be identified as ending punctuation include commas, periods, and questions); the Privacy InformationNormalizer (PII) masks personal information (such as email addresses, phone numbers, or home addresses) from text data; and the Tokenizer performs word segmentation on text data.

[0129] Classification operators are used to classify text data. For example, language classification operators are used to classify text data based on language, such as Chinese, English, or Japanese; sensitive content identification operators are used to identify text data containing sensitive content; and topic classification operators are used to classify text data based on its topics.

[0130] Scoring operators are used to score the quality of text data. For example, the repetition rate scoring operator is used to score based on the text repetition rate in the text data; the complexity scoring operator is used to score based on the complexity of the text data; and the diversity scoring operator is used to score based on the diversity of the text data.

[0131] exist Figure 7 Among the operators shown, the JSON read operator, image read operator, JSON write operator, personal information masking operator, language classification operator, and sensitive content recognition operator are operators that must be executed during the data cleaning process, while other operators are optional. Therefore, in the process of generating basic operators, the pipeline generation module can generate the operators that must be executed during the data cleaning process within the specified operator range, thereby obtaining the basic operators.

[0132] Step 3: The pipeline execution modules in the master and slave nodes execute the basic operators.

[0133] After the pipeline generation module generates basic operators, the pipeline management module can assign the various operators included in the basic operators to the pipeline execution modules in the master and slave nodes for execution, thereby completing the initial cleaning of the dataset by executing the basic operators. Specifically, the process of the pipeline execution module executing the basic operators can correspond to the process of cleaning the data in the original dataset according to the basic cleaning requirements described in the above embodiments.

[0134] It should be noted that not all operators included in the basic operators may be executed in step 3. For example, the JSON write operator may only be executed after all data cleaning steps are completed, thus writing the cleaned data back to the storage layer. Similarly, the personal information masking operator, which relies on regular expressions and is computationally intensive, may also be postponed until most of the data has been cleaned.

[0135] Step 4: The pipeline generation module negotiates with the task rule management module to determine the metric operator to be generated based on the downstream task type.

[0136] Specifically, the pipeline generation module can send the downstream task type to the task rule management module, which then determines the metric operator to be generated based on the downstream task type and feeds back the metric operator to the pipeline generation module.

[0137] Here, the metric operator is used to statistically analyze the attribute values ​​of data in the data set under various attributes (i.e., the attribute values ​​of data under the target attribute as described in the above embodiments). Therefore, the task rule management module actually determines which attributes of the data in the dataset need to be statistically analyzed based on the downstream task type (corresponding to the determination of the target attribute as described in the above embodiments).

[0138] The task rule management module has several ways to determine the metric operators to be generated based on the downstream task type.

[0139] Method 1: Determine the metric operators to be generated based on the rules already established in the rule base.

[0140] In Method 1, a mapping relationship between downstream task types and data attributes is pre-constructed manually based on expert experience or knowledge (such as research in existing papers), and this mapping relationship is stored in a rule base. Then, the task rule management module can determine the attributes corresponding to the current downstream task type by querying the mapping relationship in the rule base, and thus determine the metric operator to be generated.

[0141] Method 2: Determine the metric operators to be generated based on a pre-trained attribute prediction model.

[0142] In Method 2, the attribute prediction model is pre-trained based on historical data related to data cleaning over a historical period (such as the types of downstream tasks applied after data cleaning, the attributes used during data cleaning, and the performance of the model trained after data cleaning). Thus, the task rule management module can determine the metric operators to be generated based on the output of the attribute prediction model by inputting the downstream task types into it.

[0143] Step 5: The pipeline generation module generates the metric operator.

[0144] After obtaining the required metric operators from the task rule management module, the pipeline generation module can generate the corresponding metric operators.

[0145] Step 6: The pipeline execution module executes the metric operator to obtain statistical information of the dataset.

[0146] The measurement operator can include one or more attribute statistics operators, each of which can determine the attribute value of the data in the dataset under a certain attribute. For example, the word count operator can count the number of words in each text data, and the average line length operator can count the average number of words in each line of each text data.

[0147] Furthermore, the attribute values ​​obtained by the metric operator from the data in the dataset can be recorded by adding a metadata field to the original data, or by using a separate index file.

[0148] For example, please refer to Figure 8 , Figure 8 This is a comparative illustration of text before and after the markup provided for this application. (Example:) Figure 8 As shown, a text stored in JSON format has attribute values ​​added to its metadata field after being processed by a metrics operator. Specifically, the text before processing is: "I'm fine, thank you. But who are you?" After the metrics operator processes the text, the following content is added to the text: "word_count:9,", "average_line_length:4.5". That is, the text has 9 words, and the average number of words per line is 4.5.

[0149] Step 7: The pipeline generation module negotiates with the metric rule management module to determine the hyperparameters of the cleaning operator based on the statistical information of the dataset.

[0150] After the pipeline execution module executes the metric operator, the pipeline generation module can obtain statistical information about the dataset from the pipeline execution module and perform further statistical analysis on this information. For example, the pipeline generation module can obtain histograms, medians, maximum values, and minimum values ​​related to the number of words in each data point in the dataset; similarly, it can obtain histograms, medians, maximum values, and minimum values ​​related to the category of each data point in the dataset.

[0151] Specifically, the pipeline generation module can send the statistical information of the dataset to the metric rule management module. The metric rule management module determines the hyperparameters of the cleaning operator (corresponding to the cleaning metrics described in the above embodiments) based on the statistical information of the dataset and the downstream task type. That is, the range of attribute values ​​of the data to be cleaned indicated in the cleaning operator. The cleaning operator may include one or more operators, and different operators are used to perform data cleaning based on different attribute value ranges.

[0152] There are several ways to determine the hyperparameters of the cleaning operator.

[0153] Method 1: Determine the hyperparameters of the cleaning operator based on the rules already established in the rule base.

[0154] In Method 1, a mapping relationship between the statistical information of the dataset and the hyperparameters of the cleaning operator is pre-constructed manually based on expert experience or knowledge (such as research in existing papers), and the generated mapping relationship is stored in a rule base. Then, the metric rule management module can determine the hyperparameter combination corresponding to the statistical information of the current dataset by querying the mapping relationship in the rule base, and thus determine the hyperparameters of the cleaning operator.

[0155] Method 2: Determine the hyperparameters of the cleaning operator based on hyperparameter optimization (HPO).

[0156] Specifically, HPO refers to finding the optimal combination of hyperparameters through multiple trials, thereby determining the hyperparameters of each operator included in the cleaning operator.

[0157] Method 3: Determine the hyperparameters of the cleaning operator based on a pre-trained second artificial intelligence inference model.

[0158] In Method 3, the second AI inference model is pre-trained based on historical data related to data cleaning (such as the types of downstream tasks applied after data cleaning, statistical information of the dataset, and the performance of the model trained after data cleaning). Thus, the metric rule management module can determine the cleaning operators to be generated based on the output of the second AI inference model by inputting the types of downstream tasks and statistical information of the dataset.

[0159] Step 8: The pipeline generation module generates a cleaning operator based on the hyperparameters of the cleaning operator.

[0160] For example, the pipeline generation module can generate a word count-based cleaning operator (Word Count Filter). The Word Count Filter is set to (min = 20, max = 200), meaning it will filter out text with fewer than 20 words or more than 200 words.

[0161] Step 9: The pipeline execution module executes the cleaning operator.

[0162] During the execution of cleaning operators, the pipeline execution module can employ an operator fusion optimization method to merge multiple cleaning operators, thereby improving data cleaning efficiency. For example, please refer to [link to example]. Figure 9 , Figure 9 This is a schematic diagram illustrating an operator fusion optimization method provided in this application. For example... Figure 9 As shown, assume the original cleaning operators to be executed include a cleaning operator based on average line length (average_line_length_filter) and a cleaning operator based on maximum line length (max_line_length_filter). Before operator fusion, average_line_length_filter needs to be executed first to filter out text with an average line length of not less than 500; then, max_line_length_filter is executed to filter out text with a maximum line length of not less than 13381. Clearly, before operator fusion, the data in the dataset needs to be scanned twice to complete the operator execution. Since the attributes involved in these two operators are full-text granular, and the attributes of each data point have already been obtained when the measurement operators were executed previously, these two operators can be fused to obtain a fused operator. The fused operator can simultaneously filter the data based on both attributes, thus enabling the operator execution to be completed with only a single scan of the dataset.

[0163] Step 10: The pipeline execution module stores the cleaned dataset into the storage layer.

[0164] After the pipeline execution module has executed all the cleaning operators, the cleaned dataset is obtained. The pipeline execution module then stores the cleaned dataset in the storage layer to complete the data cleaning process.

[0165] The method provided in this application has been described in detail above. Next, the device provided in this application for performing the above method will be described.

[0166] Please see Figure 10 , Figure 10 This is a schematic diagram of a data cleaning apparatus provided in this application. Figure 10 As shown, the data cleaning device includes: an acquisition module 1001, used to acquire the task type of the task to be cleaned in the dataset; a processing module 1002, used to determine the cleaning operation to be performed on the dataset according to the task type; the processing module 1002 is also used to analyze the dataset according to the cleaning method corresponding to the cleaning operation and obtain the analysis result; the acquisition module 1001 is also used to acquire the cleaning conditions corresponding to the cleaning operation according to the analysis result; the processing module 1002 is also used to perform the cleaning operation on the data in the dataset according to the cleaning conditions.

[0167] In one possible implementation, when the processing module 1002 determines the cleaning operation for cleaning the dataset based on the task type, it specifically performs the following: inputs the task type into the first artificial intelligence inference model, and outputs the cleaning operation from the first artificial intelligence inference model.

[0168] In one possible implementation, when the acquisition module 1001 acquires the cleaning conditions corresponding to the cleaning operation based on the analysis results, it is specifically used to: input the analysis results into the second artificial intelligence inference model, and have the second artificial intelligence model output the cleaning conditions.

[0169] In one possible implementation, before the acquisition module 1001 acquires the task type of the task to be cleaned, the processing module 1002 is also used to perform a preset basic cleaning operation on the dataset.

[0170] In one possible implementation, when the processing module 1002 analyzes the dataset according to the cleaning method corresponding to the cleaning operation, it is specifically used to: determine the attribute values ​​of the data in the dataset under the target attribute according to the cleaning method corresponding to the cleaning operation, obtain the attribute value set, and the analysis result includes the attribute value set; wherein, the cleaning method includes performing cleaning on the data based on the attribute values ​​of the data under the target attribute.

[0171] In one possible implementation, the cleaning conditions include a first condition, which indicates that the value of the first attribute of the data to be cleaned is within a first range, and the target attribute includes the first attribute; when the processing module 1002 performs a cleaning operation on the data in the dataset according to the cleaning conditions, it is specifically used to: based on the set of attribute values, remove data in the dataset whose value of the first attribute is within a first range.

[0172] In one possible implementation, when the acquisition module 1001 acquires the cleaning conditions corresponding to the request cleaning operation based on the analysis results, it is specifically used to: acquire data retention conditions, which are used to indicate the conditions that the amount of data retained after the dataset is cleaned must meet; and acquire the cleaning conditions corresponding to the request cleaning operation based on the data retention conditions and the analysis results.

[0173] In one possible implementation, the task type is used to indicate the type of model training task, where the data in the dataset is text data, image data, video data, or audio data.

[0174] Both the acquisition module 1001 and the processing module 1002 can be implemented in software or in hardware. For example, the implementation of the processing module 1002 will be described below. Similarly, the implementation of the acquisition module 1001 can be referenced from the implementation of the processing module 1002.

[0175] As an example of a software functional unit, the processing module 1002 may include code running on a computing instance. The computing instance may include at least one of a physical host (computing device), a virtual machine, or a container. Further, the aforementioned computing instance may be one or more. For example, the processing module 1002 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed in the same region or in different regions. Further, the multiple hosts / virtual machines / containers used to run the code may be distributed in the same availability zone (AZ) or in different AZs, each AZ including one or more geographically proximate data centers. Typically, a region may include multiple AZs.

[0176] Similarly, multiple hosts / virtual machines / containers used to run this code can be distributed within the same Virtual Private Cloud (VPC) or across multiple VPCs. Typically, a VPC is set up within a region. Communication between two VPCs within the same region, as well as between VPCs in different regions, requires a communication gateway to be set up within each VPC to enable interconnection between VPCs.

[0177] As an example of a hardware functional unit, the processing module 1002 may include at least one computing device, such as a server. Alternatively, the processing module 1002 may be implemented using a central processing unit (CPU), an application-specific integrated circuit (ASIC), or a programmable logic device (PLD). The PLD may be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), a data processing unit (DPU), a neural network processing unit (NPU), a system-on-chip (SoC), an offload card, an accelerator card, or any combination thereof.

[0178] The processing module 1002 includes multiple computing devices that can be distributed within the same region or in different regions. Similarly, the processing module 1002 can be distributed within the same Availability Zone (AZ) or in different AZs. Likewise, the processing module 1002 can be distributed within the same Virtual Private Cloud (VPC) or in multiple VPCs. These multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, GALs, DPUs, NPUs, SoCs, offloading cards, and accelerator cards.

[0179] Please see Figure 11 , Figure 11 A schematic diagram of the structure of a computing device provided in this application. Figure 11The computing device 1100 shown can be used to perform the data cleaning method provided in this embodiment. For example... Figure 11 As shown, the computing device 1100 includes a bus 1102, a processor 1104, a memory 1106, and a communication interface 1108. The processor 1104, the memory 1106, and the communication interface 1108 communicate with each other via the bus 1102. The computing device 1100 can be a server or a terminal device. It should be understood that this application does not limit the number of processors and memories in the computing device 1100.

[0180] Bus 1102 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, Figure 11 The bus 1102 may be represented by a single line, but this does not mean that there is only one bus or one type of bus. The bus 1102 may include a path for transmitting information between various components of the computing device 1100 (e.g., memory 1106, processor 1104, communication interface 1108).

[0181] The processor 1104 may include any one or more processors such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0182] The memory 1106 may include volatile memory, such as random access memory (RAM). The processor 1104 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD).

[0183] The memory 1106 stores executable program code, and the processor 1104 executes this executable program code to implement the functions of the aforementioned acquisition module and processing module, thereby realizing the data cleaning method described above. That is, the memory 1106 stores instructions for executing the data cleaning method.

[0184] The communication interface 1108 uses transceiver modules such as, but not limited to, network interface cards and transceivers to enable communication between the computing device 1100 and other devices or communication networks.

[0185] This application also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smartphone.

[0186] Please see Figure 12 , Figure 12 This is a schematic diagram of a computing device cluster provided in this application. Figure 12 As shown, the computing device cluster includes at least one computing device 1100. The memory 1106 of one or more computing devices 1100 in the computing device cluster may store the same instructions for performing data cleaning methods.

[0187] In some possible implementations, the memory 1106 of one or more computing devices 1100 in the computing device cluster may also store partial instructions for executing the data cleaning method. In other words, a combination of one or more computing devices 1100 can jointly execute the instructions for executing the data cleaning method.

[0188] It should be noted that the memory 1106 in different computing devices 1100 within the computing device cluster can store different instructions, each used to execute a portion of the functions of the data cleaning method apparatus. That is, the instructions stored in the memory 1106 of different computing devices 1100 can implement the functions of one or more of the aforementioned acquisition and processing modules.

[0189] In some possible implementations, one or more computing devices in a computing device cluster can be connected via a network. This network can be a wide area network (WAN), a local area network (LAN), or similar. Figure 13 One possible implementation method is shown. Figure 13 This is a schematic diagram of another computing device cluster structure provided in this application. Figure 13 As shown, in the computing device cluster 1300, two computing devices 1100A and 1100B are connected via a network. Specifically, they are connected to the network through the communication interfaces in each computing device. In this possible implementation, the memory 1106 in computing device 1100A stores instructions for executing the functions of the acquisition module. Simultaneously, the memory 1106 in computing device 1100B stores instructions for executing the functions of the processing module.

[0190] It should be understood that Figure 13 The functions of computing device 1100A shown can also be performed by multiple computing devices 1100. Similarly, the functions of computing device 1100B can also be performed by multiple computing devices 1100.

[0191] This application also provides a chip comprising a processing unit and a communication unit. The processing unit may be, for example, a processor, and the communication unit may be, for example, an input / output interface, pins, or circuits. The processing unit can execute computer execution instructions stored in a storage unit to cause the chip within the electronic device to perform the methods described in the above embodiments. Optionally, the storage unit can be an on-chip storage unit, such as a register or cache. Alternatively, the storage unit can be an external storage unit located within a wireless access device, such as read-only memory (ROM) or other types of static storage devices capable of storing static information and instructions, such as random access memory (RAM).

[0192] Please refer to Figure 14 , Figure 14 This is a schematic diagram of a computer-readable storage medium provided in this application. This application also provides a computer-readable storage medium in some embodiments, wherein the above-described... Figure 2 The disclosed method can be implemented as computer program instructions encoded in a machine-readable format on a computer-readable storage medium or on other non-transitory media or articles of art.

[0193] Figure 14 A conceptual partial view of an example computer-readable storage medium arranged according to at least some of the embodiments shown herein is illustrated schematically. The example computer-readable storage medium includes a computer program for executing computer processes on a computing device.

[0194] In one embodiment, the computer-readable storage medium 1400 is provided using a signal bearer medium 1401. The signal bearer medium 1401 may include one or more program instructions 1402, which, when executed by one or more processors, can provide the above-mentioned... Figure 2 The described function or part of the function.

[0195] In some examples, signal carrying medium 1401 may include computer-readable medium 1403, such as, but not limited to, hard disk drive, compact disc (CD), digital video disc (DVD), digital magnetic tape, memory, ROM or RAM, etc.

[0196] In some embodiments, the signal-bearing medium 1401 may include a computer-recordable medium 1404, such as, but not limited to, a memory, a read / write (R / W) CD, a R / W DVD, etc. In some embodiments, the signal-bearing medium 1401 may include a communication medium 1405, such as, but not limited to, digital and / or analog communication media (e.g., fiber optic cables, waveguides, wired communication links, wireless communication links, etc.). Therefore, for example, the signal-bearing medium 1401 may be transmitted by a wireless communication medium 1405 (e.g., a wireless communication medium conforming to the IEEE 1102.X standard or other transmission protocols).

[0197] One or more program instructions 1402 may be, for example, computer-executable instructions or logical implementation instructions. In some examples, the computing device may be configured to provide various operations, functions, or actions in response to one or more program instructions 1402 conveyed to the computing device via a computer-readable medium 1403, a computer-recordable medium 1404, and / or a communication medium 1405.

[0198] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the accompanying drawings of the device embodiments provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0199] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CPUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk, or optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, training equipment, or network device, etc.) to execute the methods of the various embodiments of this application.

[0200] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product.

[0201] A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions according to this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another. For example, computer instructions can be transferred from one website, computer, training device, or data center to another website, computer, training device, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can store or a data storage device such as a training device or data center that integrates one or more available media. The available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives (SSDs)).

Claims

1. A data cleaning method, characterized in that, include: Obtain the task type for processing the dataset to be cleaned; The cleaning operation for cleaning the dataset is determined based on the task type. The dataset is analyzed according to the cleaning method corresponding to the cleaning operation to obtain the analysis results; Based on the analysis results, obtain the cleaning conditions corresponding to the cleaning operation. The cleaning operation is performed on the data in the dataset according to the cleaning conditions.

2. The method according to claim 1, characterized in that, The cleaning operations for cleaning the dataset based on the task type include: The task type is input into the first artificial intelligence inference model, and the first artificial intelligence inference model outputs the cleaning operation.

3. The method according to claim 1 or 2, characterized in that, The step of obtaining the cleaning conditions corresponding to the cleaning operation based on the analysis results includes: The analysis results are input into the second artificial intelligence inference model, and the second artificial intelligence model outputs the cleaning conditions.

4. The method according to any one of claims 1-3, characterized in that, Before obtaining the task type of the task processing the dataset to be cleaned, the method further includes: Perform a preset basic cleaning operation on the dataset.

5. The method according to any one of claims 1-4, characterized in that, The step of analyzing the dataset according to the cleaning method corresponding to the cleaning operation includes: Based on the cleaning method corresponding to the cleaning operation, the attribute values ​​of the data in the dataset under the target attribute are determined to obtain the attribute value set, and the analysis result includes the attribute value set. The cleaning method includes cleaning the data based on the attribute values ​​of the data under the target attribute.

6. The method according to claim 5, characterized in that, The cleaning conditions include a first condition, which indicates that the value of a first attribute of the data to be cleaned is within a first range, and the target attribute includes the first attribute. Performing the cleaning operation on the data in the dataset according to the cleaning conditions includes: Based on the set of attribute values, data in the dataset whose values ​​of the first attribute fall within the first range are removed.

7. The method according to any one of claims 1-6, characterized in that, The step of obtaining the cleaning conditions corresponding to the washing operation based on the analysis results includes: Obtain data retention conditions, which are used to indicate the conditions that the amount of data retained after the dataset is cleaned must meet; Based on the data retention conditions and the analysis results, the cleaning conditions corresponding to the requested washing operation are obtained.

8. The method according to any one of claims 1-7, characterized in that, The task type is used to indicate the type of model training task, and the data in the dataset is text data, image data, video data, or audio data.

9. A data cleaning apparatus, characterized in that, include: The acquisition module is used to obtain the task type of the task that processes the dataset to be cleaned; The processing module is used to determine the cleaning operation when cleaning the dataset according to the task type; The processing module is also used to analyze the dataset according to the cleaning method corresponding to the cleaning operation, and obtain the analysis results; The acquisition module is also used to acquire the cleaning conditions corresponding to the washing operation based on the analysis results; The processing module is further configured to perform the cleaning operation on the data in the dataset according to the cleaning conditions.

10. The apparatus according to claim 9, characterized in that, When the processing module determines the cleaning operation for cleaning the dataset based on the task type, it is specifically used for: The task type is input into the first artificial intelligence inference model, and the first artificial intelligence inference model outputs the cleaning operation.

11. The apparatus according to claim 9 or 10, characterized in that, When the acquisition module obtains the cleaning conditions corresponding to the wash request operation based on the analysis results, it is specifically used for: The analysis results are input into the second artificial intelligence inference model, and the second artificial intelligence model outputs the cleaning conditions.

12. A computing device cluster, characterized in that, It includes at least one computing device, each computing device including a processor and memory; The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device to cause the cluster of computing devices to perform the operational steps of the method as described in any one of claims 1 to 8.

13. A computer storage medium, characterized in that, The computer storage medium stores instructions that, when executed by the computer, cause the computer to perform the method according to any one of claims 1 to 8.

14. A computer program product, characterized in that, The computer program product stores instructions that, when executed by a computer, cause the computer to perform the method according to any one of claims 1 to 8.

Images