Large model-oriented data cleaning method and device

By dividing the large data set into data blocks and processing them in parallel at multiple nodes, combining priority order and multi-threaded processing, the problem of low efficiency of large-scale data cleaning in the existing technology is solved, and efficient data cleaning and resource optimization are achieved , improves the model training effect.

CN118916353APending Publication Date: 2024-11-08SHANDONG LANGCHAO YUNTOU INFORMATION TECH CO LTD
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
CN202411159164.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-22
Publication Date
2024-11-08

AI Technical Summary

Technical Problem

The existing data cleaning methods have low performance when processing large-scale data sets and are centered on cleaning strategies. They are not suitable for large-scale data set cleaning, resulting in low overall data cleaning efficiency.

Method used

By dividing the data set to be cleaned into multiple data blocks, and matching the data running nodes according to the task resource configuration and the remaining resources of the cluster, multiple cleaning strategies are applied in order of priority to clean the data blocks, and data rows are processed in parallel by using multiple threads in parallel , monitor and merge metric data in real time.

Benefits of technology

The cleaning efficiency of large model data sets is improved, multiple scans of data sets to be cleaned are avoided, the running time is significantly reduced, the data-centric cleaning effect is achieved, resource utilization is optimized, and model training effect and prediction accuracy are improved.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118916353A_ABST
    Figure CN118916353A_ABST
Patent Text Reader

Abstract

The invention discloses a data cleaning method and device for a large model, and the method comprises the steps: determining a corresponding cleaning task based on a to-be-cleaned data set of the large model; segmenting the to-be-cleaned data set to obtain a plurality of data blocks; determining a data operation node matched with each data block according to the task resource configuration and current remaining resources of the cluster; based on the cleaning strategy configuration, generating a data cleaning operation corresponding to each data block; distributing each data block and the corresponding data cleaning operation to a data operation node matched with each data block, and triggering each data operation node to perform data cleaning on the data blocks by adopting a plurality of cleaning strategies in sequence according to a priority sequence from high to low so as to obtain cleaned data; based on the plurality of cleaned data, valid data required for the large model is determined. According to the method, it can be ensured that the cleaning object facing each cleaning strategy is the data block, multiple times of scanning of the to-be-cleaned data set are avoided, and therefore the data cleaning efficiency is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of big data processing, and in particular to a data cleaning method and device for large models. Background Art

[0002] Data cleaning is an important part of big data processing and large model data engineering. The quality of data cleaning directly affects the training effect and prediction accuracy of the model. Therefore, how to efficiently clean data is an important issue facing big data processing and model training technology. Existing data cleaning methods mainly clean the data set for each cleaning strategy. Each cleaning strategy cleans the data, counts monitoring indicators, saves the cleaned data set, and then processes the subsequent cleaning strategies in turn. This cleaning strategy-centered cleaning facilitates independent processing and test monitoring of the strategy, and is usually more suitable for small-scale data sets.

[0003] However, when the data set is very large, existing data cleaning methods have some problems. First, multiple cleaning strategies require full scanning and multiple passes of the data set, which will result in very low operating performance. In addition, existing methods are centered on cleaning strategies and are not suitable for cleaning data sets for large models, resulting in low overall data cleaning efficiency. Summary of the invention

[0004] The present application provides a data cleaning method and device for large models, aiming to improve the cleaning efficiency of data sets of large models.

[0005] In order to achieve the above objectives, this application provides the following technical solutions:

[0006] A data cleaning method for a large model, comprising:

[0007] Based on the data set to be cleaned in the large model, a corresponding cleaning task is determined; the cleaning task includes task resource configuration and cleaning strategy configuration; the cleaning strategy configuration includes multiple cleaning strategies and corresponding priorities;

[0008] Segmenting the data set to be cleaned to obtain multiple data blocks;

[0009] Determine the data running node that matches each of the data blocks according to the task resource configuration and the current remaining resources of the cluster;

[0010] Based on the cleaning strategy configuration, generating a data cleaning job corresponding to each of the data blocks;

[0011] Distribute each of the data blocks and the corresponding data cleaning job to the data operation node matched by each of the data blocks, and trigger each of the data operation nodes to sequentially use the multiple cleaning strategies to clean the data blocks in descending order of priority to obtain the corresponding cleaned data;

[0012] Based on the obtained plurality of cleaned data, valid data required for the large model is determined.

[0013] Optionally, the process of triggering the data operation node to sequentially use a plurality of the cleaning strategies to clean the data block in descending order of priority includes:

[0014] The data operation node is triggered to split the data block into multiple data rows, and multiple threads are used to perform a target operation on the multiple data rows in parallel to obtain multiple cleaned data rows; the target operation is: in order of priority from high to low, multiple cleaning strategies are used in sequence to clean the data rows.

[0015] Optionally, the method further includes:

[0016] Obtain monitoring indicator data of each of the data operation nodes and display each of the monitoring indicator data; the monitoring indicator data includes monitoring indicators of multiple cleaning strategies and corresponding indicator values; the indicator values ​​are collected in real time based on the data operation node calling a preset core class.

[0017] Optionally, the method further includes:

[0018] The monitoring index data of each data operation node when the cleaning task is completed are merged to obtain unified index data of multiple cleaning strategies when the cleaning task is completed; the unified index data includes the monitoring index of the cleaning strategy and the corresponding effective index value; the effective index value is the sum of the index values ​​shown in each monitoring index data.

[0019] A data cleaning device for large models, comprising:

[0020] A task determination unit, used to determine a corresponding cleaning task based on the to-be-cleaned data set of the large model; the cleaning task includes task resource configuration and cleaning strategy configuration; the cleaning strategy configuration includes multiple cleaning strategies and corresponding priorities;

[0021] A data segmentation unit, used for segmenting the data set to be cleaned to obtain a plurality of data blocks;

[0022] A node matching unit, used to determine a data running node that matches each data block according to the task resource configuration and the current remaining resources of the cluster;

[0023] A job generating unit, configured to generate a data cleaning job corresponding to each of the data blocks based on the cleaning strategy configuration;

[0024] A job execution unit is used to distribute each of the data blocks and the corresponding data cleaning job to the data operation node matched by each of the data blocks, triggering each of the data operation nodes to sequentially use the multiple cleaning strategies to clean the data blocks in descending order of priority, so as to obtain the corresponding cleaned data;

[0025] The data integration unit is used to determine the valid data required for the large model based on the obtained multiple cleaned data.

[0026] Optionally, the process in which the job execution unit triggers the data operation node to sequentially use the multiple cleaning strategies to clean the data block in order of priority from high to low includes:

[0027] The data operation node is triggered to split the data block into multiple data rows, and multiple threads are used to perform a target operation on the multiple data rows in parallel to obtain multiple cleaned data rows; the target operation is: in order of priority from high to low, multiple cleaning strategies are used in sequence to clean the data rows.

[0028] Optionally, the device further comprises:

[0029] A cleaning monitoring unit is used to obtain monitoring indicator data of each data operation node and display each monitoring indicator data; the monitoring indicator data includes monitoring indicators of multiple cleaning strategies and corresponding indicator values; the indicator values ​​are collected in real time based on the data operation node calling the preset core class.

[0030] Optionally, the cleaning monitoring unit is further used for:

[0031] The monitoring index data of each data operation node when the cleaning task is completed are merged to obtain unified index data of multiple cleaning strategies when the cleaning task is completed; the unified index data includes the monitoring index of the cleaning strategy and the corresponding effective index value; the effective index value is the sum of the index values ​​shown in each monitoring index data.

[0032] A storage medium includes a stored program, wherein the program executes the large model-oriented data cleaning method when executed by a processor.

[0033] An electronic device, comprising: a processor, a memory and a bus; the processor and the memory are connected via the bus;

[0034] The memory is used to store programs, and the processor is used to run programs, wherein the program executes the large model-oriented data cleaning method when the processor runs it.

[0035] The technical solution provided by the present application determines the corresponding cleaning task based on the data set to be cleaned of the large model. The data set to be cleaned is divided to obtain multiple data blocks. According to the task resource configuration and the current remaining resources of the cluster, the data operation node matched by each data block is determined. Based on the cleaning strategy configuration, a data cleaning job corresponding to each data block is generated. Each data block and the corresponding data cleaning job are distributed to the data operation node matched by each data block, and each data operation node is triggered to use multiple cleaning strategies to clean the data block in order from high to low priority to obtain the corresponding cleaned data. Based on the multiple cleaned data obtained, it is determined to be the valid data required for the large model. The present application can ensure that the cleaning object faced by each cleaning strategy is the data block (data row) rather than the entire data set to be cleaned, so as to achieve a data-centric cleaning effect and avoid multiple scans of the data set to be cleaned, thereby effectively improving the cleaning efficiency of the data set of the large model. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0037] Figure 1 A schematic diagram of a process flow of a data cleaning method for a large model provided in an embodiment of the present application;

[0038] Figure 2 A flowchart of another data cleaning method for large models provided in an embodiment of the present application;

[0039] Figure 3 A schematic diagram of the architecture of a data cleaning system provided in an embodiment of the present application;

[0040] Figure 4 A core class design architecture diagram provided for an embodiment of the present application;

[0041] Figure 5 A schematic diagram of the architecture of a data cleaning device for large models provided in an embodiment of the present application. DETAILED DESCRIPTION

[0042] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0043] In this application, relational terms such as first and second, etc. are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. The terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of more restrictions, the elements defined by the sentence "comprise one..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0044] like Figure 1 As shown, it is a flow chart of a data cleaning method for a large model provided in an embodiment of the present application, which includes the following steps.

[0045] S101: Determine the corresponding cleaning task based on the to-be-cleaned data set of the large model.

[0046] The cleaning task includes task resource configuration and cleaning strategy configuration. The cleaning strategy configuration includes multiple cleaning strategies and corresponding priorities.

[0047] The so-called task resource configuration can be understood as the computing resources required for data cleaning of the data set to be cleaned. Generally speaking, the task resource configuration can be determined according to the data size of the data set to be cleaned.

[0048] The so-called cleaning resource configuration can be understood as the serial assembly of multiple cleaning strategies used when cleaning the data set to be cleaned to determine the priorities of multiple cleaning strategies. The priority determines the execution order of the cleaning strategies when cleaning the data set to be cleaned. Generally speaking, a cleaning strategy with a higher priority will clean the data set to be cleaned in advance compared to a cleaning strategy with a lower priority. A cleaning strategy with a lower priority will clean the data set to be cleaned again after it has been cleaned by a cleaning strategy with a higher priority.

[0049] It should be noted that the cleaning strategy configuration can be set by the technician according to the actual situation.

[0050] In some examples, the cleaning strategies include, but are not limited to: similarity deduplication strategy, erroneous data removal strategy, text dirty data strategy, etc.

[0051] S102: Segment the data set to be cleaned to obtain multiple data blocks.

[0052] After the cleaning task corresponding to the data set to be cleaned is determined, the data set to be cleaned may be segmented according to the data size of the data set to be cleaned to obtain multiple data blocks.

[0053] In some examples, the currently available node resources of the cluster (such as the number of available data operation nodes in the cluster) can also be used as a reference to ensure that the multiple data blocks obtained by segmenting the data set to be cleaned can be evenly distributed to the multiple data operation nodes available in the cluster.

[0054] In a possible implementation, the Spark framework on the Hadoop platform may be used to segment the data set to be cleaned to obtain multiple data blocks.

[0055] S103: Determine the data running node that matches each data block according to the task resource configuration and the current remaining resources of the cluster.

[0056] Among them, according to the task resource configuration and the current remaining resources of the cluster, the data operation node matching each data block is determined, which can achieve load balancing among multiple data operation nodes.

[0057] It should be noted that, based on the task resource configuration and the current remaining resources of the cluster, as a reference for the matching relationship between data blocks and data operation nodes, appropriate resource scheduling can be implemented on the basis of the relatively limited current remaining resources of the cluster, ensuring that each data operation node has sufficient resources to process the corresponding data blocks, so that the cleaning task can be effectively completed.

[0058] S104: Generate a data cleaning job corresponding to each data block based on the cleaning strategy configuration.

[0059] The data cleaning job is used to clean the data blocks using the cleaning strategy configuration.

[0060] S105: Distribute each data block and the corresponding data cleaning job to the data operation node matched by each data block, trigger each data operation node to use multiple cleaning strategies in descending order of priority to clean the data block, so as to obtain the corresponding cleaned data.

[0061] Among them, each data operation node performs data cleaning on the allocated data block to obtain the cleaned data corresponding to the data block, which can significantly improve the efficiency of data cleaning.

[0062] Optionally, the process of triggering the data running node to use multiple cleaning strategies in order of high to low priority to clean the data block may include: triggering the data running node to split the data block into multiple data rows, and using multiple threads to perform a target operation on the multiple data rows in parallel to obtain multiple cleaned data rows; the target operation is: using multiple cleaning strategies in order of high to low priority to clean the data rows.

[0063] It should be noted that a plurality of cleaned data rows constitute the corresponding cleaned data. In addition, splitting the data block into a plurality of data rows and triggering the data operation node to use a plurality of threads to perform the target operation on the plurality of data rows in parallel to obtain a plurality of cleaned data rows can effectively improve the data cleaning efficiency of the data block.

[0064] Generally speaking, multiple cleaning strategies are used in order of priority from high to low to clean the data rows. This can be understood as: with data as the center, multiple cleaning strategies are serially assembled to clean the data. The cleaning strategy is aimed at the data row each time, rather than the entire data set.

[0065] In addition, when the data operation node adopts the cleaning strategy configuration to clean the data block, the data cleaning effect of each cleaning strategy can also be monitored, and the data cleaning effect can be reflected through the preset monitoring indicators.

[0066] Optionally, you can monitor the data cleaning effect of each cleaning strategy, see Figure 2 Steps shown and explanations of the steps.

[0067] S106: Based on the obtained multiple cleaned data, determine the valid data required for the large model.

[0068] After each data operation node completes the cleaning of the data block, it will also release resources so that subsequent new cleaning tasks can obtain available resources from the resource pool. After all data blocks have been cleaned by the data operation node and the cleaned data corresponding to all data blocks have been obtained, the cleaning task is determined to be completed, and based on multiple cleaned data, it is determined to be valid data required for the large model.

[0069] In some examples, combined Figure 2 The method shown can be used to design a corresponding data cleaning system. The framework of the data cleaning system is as follows Figure 3 As shown, it includes a data management unit, a resource allocation unit and a data operation node.

[0070] Specifically, the data management unit is mainly responsible for cleaning task management, data splitting, and the collection and display of monitoring indicator data.

[0071] Cleaning task management includes life cycle management and dynamic real-time adjustment of cleaning strategies.

[0072] Lifecycle management includes cleaning task configuration creation (such as task resource configuration, cleaning strategy configuration, etc.), task running, task destruction, etc.

[0073] Dynamic real-time adjustment of the cleaning strategy means dynamically adjusting the cleaning strategy during the task cleaning process. According to the changes in the data set to be cleaned, the cleaning requirements, and the monitoring indicator data, the number of cleaning strategies can be increased or decreased, the parameters of the cleaning strategy can be modified, etc. After each cleaning strategy adjustment, it will be re-sent to each data operation node of the cleaning task.

[0074] Data splitting is to split the data set to be cleaned into multiple data blocks. The cleaning task divides the data set to be cleaned into appropriate data blocks according to the size of the data set to be cleaned and the data node resources.

[0075] The collection and display of monitoring indicator data is responsible for collecting and displaying the monitoring indicator data of each data operation node, so as to realize the real-time monitoring of the cleaning process effect of each cleaning strategy, help users to timely analyze and discover problems in cleaning, and provide a basis for adjusting the cleaning strategy.

[0076] Specifically, the resource allocation unit is mainly responsible for data node resource allocation and sending data blocks to data nodes. Specifically, according to the task resource configuration of the cleaning task and the data blocks split by the data set to be cleaned, and according to the current remaining resources of the cluster, the resource allocation unit will make appropriate allocation and resource scheduling based on the remaining resources of the existing cluster nodes, and send each data block to each data operation node of the distributed cluster to ensure that the distributed cluster can use the allocated resources to clean the sent data blocks. After the cleaning task is completed, the allocated resources will be released and recycled to the resource pool so that new cleaning tasks can apply for allocation.

[0077] Specifically, the data operation node is mainly responsible for executing data cleaning tasks in parallel and storing the cleaned data to the large model. On each data operation node, the data block sent down will be further subdivided into multiple data rows. The data rows can be cleaned in parallel using multiple threads. Each data row is passed to the serialized assembled cleaning strategy for cleaning in turn. After the entire data block is cleaned by all cleaning strategies on the data node, the data is saved for use by the large model. The design of the cleaning strategy can be carried out according to actual needs. For example, highly similar and repeated data can be removed according to similarity, erroneous data can be corrected, and unnecessary dirty data can be filtered out. The execution of the cleaning strategy can be implemented in Python, and the Python Pandas library is used to perform data cleaning in parallel on Spark. During the cleaning process, the monitoring indicators of each cleaning strategy are recorded, such as the cleaning speed and cleaning accuracy. These monitoring indicators can be fed back to the data management unit in real time for display, so that users can understand the progress and effect of cleaning and dynamically adjust the cleaning strategy. When all data rows have been processed by the cleaning strategy, the monitoring indicator data of the cleaned data are merged, and then the monitoring indicators of each cleaning strategy are counted.

[0078] Compared with the existing technology, the data cleaning system can achieve the following beneficial effects: First, it can improve the efficiency of data cleaning. The data cleaning system uses a data-centric, serialized assembly cleaning strategy method to distribute massive data in blocks to distributed cluster nodes for parallel processing during a single traversal scanning of the data set, greatly improving the efficiency of data cleaning, especially when processing large-scale data sets, which can significantly reduce the running time; Second, it can achieve real-time monitoring of the cleaning process. The data cleaning system records monitoring indicators during the cleaning process, which can monitor the cleaning effect in real time and discover and solve problems in a timely manner, unlike the existing methods that process the complete data set in the cleaning strategy. The monitoring results can only be obtained after all strategies are processed or the final monitoring results can only be obtained after all strategies are processed; third, optimize the utilization of resources. In the data cleaning system, each cluster node passes the data row to the serialized cleaning strategy for cleaning. The cleaning strategy faces one row of original data instead of the entire data set each time. This can make more reasonable use of computing resources and avoid the problem that multiple strategies in the existing method need to fully scan and clean the data set multiple times, resulting in very low operating performance; fourth, provide training effects for large models. Since the data cleaning system can efficiently clean data and ensure the accuracy and consistency of data, it can improve the effect of model training and prediction accuracy.

[0079] The process shown in S101-S106 above splits the data set to be cleaned into multiple data blocks, uses multiple data running nodes to adopt cleaning strategy configuration, and cleans each data block to obtain the corresponding cleaned data. Compared with the existing technology, the cleaning object of each cleaning strategy is the data block (data row) rather than the entire data set to be cleaned, which can achieve a data-centric cleaning effect and avoid multiple scans of the data set to be cleaned, thereby effectively improving the cleaning efficiency of the data set of the large model.

[0080] like Figure 2 As shown, it is a flow chart of another data cleaning method for large models provided in an embodiment of the present application, which includes the following steps.

[0081] S201: Obtain monitoring indicator data of each data operation node and display each monitoring indicator data.

[0082] The monitoring indicator data includes monitoring indicators of multiple cleaning strategies and corresponding indicator values, and the indicator values ​​are collected in real time based on the data operation node calling the preset core class.

[0083] In some examples, monitoring indicators of the cleaning strategy include, but are not limited to: the number of modified data rows, the percentage of modified data, whether the data is modified, the cleaning speed, the cleaning accuracy, etc.

[0084] In some examples, the default core class design architecture can be seen in Figure 4 As shown, the preset core classes include the DataClean class. The DataClean class is responsible for performing data cleaning using the serialized assembled cleaning strategy on the data running node, and statistically summarizing the monitoring indicators of each cleaning strategy.

[0085] exist Figure 4In the content shown, dataset represents the data set to be cleaned, and cleanDataset represents the cleaned data. StrategySeries represents the serialized cleaning strategy, which consists of a series of strategy sets. You can dynamically add or delete strategies in the serialized strategy set and modify strategy parameters. strategyList represents the cleaning strategy list (i.e., cleaning strategy configuration), and adjust represents the adjustment of the cleaning strategy. PandasUDF is a custom function class based on Spark implemented in Python. It uses the Pandas library to clean data rows in parallel and records the monitoring indicators of the cleaning strategy in real time when each data row is cleaned. CleanStrategyBase represents the basic interface of the cleaning strategy, which defines the standard interface for cleaning row data. The specific implementation is performed by cleaning strategies such as the similarity deduplication strategy DedupStrategy, the error data removal strategy RemoveErrStrategy, and the text dirty data strategy TextDirtyStrategy.

[0086] S202: Merging the monitoring index data of each data operation node when the cleaning task is completed to obtain unified index data of multiple cleaning strategies when the cleaning task is completed.

[0087] The unified indicator data includes the monitoring indicators of the cleaning strategy and the corresponding effective indicator values, and the effective indicator value is the sum of the indicator values ​​shown in the various monitoring indicator data.

[0088] In some examples, unified indicator data based on the cleaning strategy can help users intuitively understand the data cleaning effect of the cleaning strategy, so that users can adaptively adjust the cleaning strategy.

[0089] The process shown in S201-S202 above can effectively monitor the data cleaning effect of each cleaning strategy on the data block by using the monitoring indicator data of each data operation node.

[0090] like Figure 5 As shown, it is a schematic diagram of the architecture of a data cleaning device for large models provided in an embodiment of the present application, including the units shown below.

[0091] The task determination unit 100 is used to determine the corresponding cleaning task based on the to-be-cleaned data set of the large model; the cleaning task includes task resource configuration and cleaning strategy configuration; the cleaning strategy configuration includes multiple cleaning strategies and corresponding priorities.

[0092] The data segmentation unit 200 is used to segment the data set to be cleaned to obtain multiple data blocks.

[0093] The node matching unit 300 is used to determine the data running node that matches each data block according to the task resource configuration and the current remaining resources of the cluster.

[0094] The job generating unit 400 is used to generate a data cleaning job corresponding to each data block based on the cleaning strategy configuration.

[0095] The job execution unit 500 is used to distribute each data block and the corresponding data cleaning job to the data operation node matched by each data block, triggering each data operation node to use multiple cleaning strategies in order of priority from high to low to perform data cleaning on the data block to obtain the corresponding cleaned data.

[0096] The data integration unit 600 is used to determine valid data required for the large model based on the obtained multiple cleaned data.

[0097] Optionally, the job execution unit 500 triggers the data running node to use multiple cleaning strategies in order of priority from high to low to perform data cleaning on the data block, including: triggering the data running node to split the data block into multiple data rows, and using multiple threads to perform target operations on the multiple data rows in parallel to obtain multiple cleaned data rows; the target operation is: using multiple cleaning strategies in order of priority from high to low to perform data cleaning on the data rows.

[0098] The cleaning monitoring unit 700 is used to obtain the monitoring index data of each data operation node and display each monitoring index data; the monitoring index data includes the monitoring indexes of multiple cleaning strategies and the corresponding index values; the index values ​​are collected in real time based on the data operation node calling the preset core class.

[0099] Optionally, the cleaning monitoring unit 700 is also used to: merge the monitoring index data of each data running node when the cleaning task is completed to obtain unified index data of multiple cleaning strategies when the cleaning task is completed; the unified index data includes the monitoring index of the cleaning strategy and the corresponding effective index value; the effective index value is the sum of the index values ​​shown in each monitoring index data.

[0100] The data cleaning device for large models shown above divides the data set to be cleaned into multiple data blocks, uses multiple data operation nodes to adopt cleaning strategy configuration, and cleans each data block to obtain corresponding cleaned data. Compared with the existing technology, the cleaning object of each cleaning strategy is the data block (data row) rather than the entire data set to be cleaned, which can achieve a data-centric cleaning effect and avoid multiple scans of the data set to be cleaned, thereby effectively improving the cleaning efficiency of the data set of large models.

[0101] The present application also provides a computer-readable storage medium, which includes a stored program, wherein the program executes the large model-oriented data cleaning method provided by the present application.

[0102] The present application also provides an electronic device, including: a processor, a memory and a bus. The processor and the memory are connected via the bus, the memory is used to store programs, and the processor is used to run the programs, wherein the data cleaning method for large models provided by the present application is executed when the programs are run.

[0103] In addition, the functions described above in the embodiments of the present application may be at least partially performed by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chip (SOCs), complex programmable logic devices (CPLDs), and the like.

[0104] Although the subject matter has been described in language specific to structural features and / or methodological logical actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims.

[0105] Although several specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present application. Certain features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination.

[0106] The above description is only a preferred embodiment of the present application and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present application is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in this application (but not limited to) by each other to form a technical solution.

Claims

1. A data cleaning method for large models, characterized in that: include: Determine the corresponding cleaning task based on the data set to be cleaned based on the large model; The cleaning task includes task resource configuration and cleaning strategy configuration; The cleaning strategy configuration includes multiple cleaning strategies and corresponding priorities; Segmenting the data set to be cleaned to obtain multiple data blocks; Determine the data running node that matches each of the data blocks according to the task resource configuration and the current remaining resources of the cluster; Based on the cleaning strategy configuration, generating a data cleaning job corresponding to each of the data blocks; Distribute each of the data blocks and the corresponding data cleaning job to the data operation node matched by each of the data blocks, and trigger each of the data operation nodes to sequentially use the multiple cleaning strategies to clean the data blocks in descending order of priority to obtain the corresponding cleaned data; Based on the obtained plurality of cleaned data, valid data required for the large model is determined.

2. The method according to claim 1, characterized in that The process of triggering the data operation node to sequentially use the plurality of cleaning strategies to clean the data block in order of priority from high to low includes: The data operation node is triggered to split the data block into multiple data rows, and multiple threads are used to perform a target operation on the multiple data rows in parallel to obtain multiple cleaned data rows; the target operation is: in order of priority from high to low, multiple cleaning strategies are used in sequence to clean the data rows.

3. The method according to claim 1, characterized in that The method further comprises: Obtain monitoring indicator data of each of the data operation nodes and display each of the monitoring indicator data; the monitoring indicator data includes monitoring indicators of multiple cleaning strategies and corresponding indicator values; the indicator values ​​are collected in real time based on the data operation node calling a preset core class.

4. The method according to claim 3, characterized in that The method further comprises: The monitoring index data of each data operation node when the cleaning task is completed are merged to obtain unified index data of multiple cleaning strategies when the cleaning task is completed; the unified index data includes the monitoring index of the cleaning strategy and the corresponding effective index value; the effective index value is the sum of the index values ​​shown in each monitoring index data.

5. A data cleaning device for large models, characterized in that: include: A task determination unit, used to determine a corresponding cleaning task based on the data set to be cleaned in the large model; The cleaning task includes task resource configuration and cleaning strategy configuration; The cleaning strategy configuration includes multiple cleaning strategies and corresponding priorities; A data segmentation unit, used for segmenting the data set to be cleaned to obtain a plurality of data blocks; A node matching unit, used to determine a data running node that matches each data block according to the task resource configuration and the current remaining resources of the cluster; A job generating unit, configured to generate a data cleaning job corresponding to each of the data blocks based on the cleaning strategy configuration; A job execution unit is used to distribute each of the data blocks and the corresponding data cleaning job to the data operation node matched by each of the data blocks, triggering each of the data operation nodes to sequentially use the multiple cleaning strategies to clean the data blocks in descending order of priority, so as to obtain the corresponding cleaned data; The data integration unit is used to determine the valid data required for the large model based on the obtained multiple cleaned data.

6. The device according to claim 5, characterized in that The process in which the job execution unit triggers the data operation node to sequentially use the multiple cleaning strategies to clean the data blocks in order of priority from high to low includes: The data operation node is triggered to split the data block into multiple data rows, and multiple threads are used to perform a target operation on the multiple data rows in parallel to obtain multiple cleaned data rows; the target operation is: in order of priority from high to low, multiple cleaning strategies are used in sequence to clean the data rows.

7. The device according to claim 5, characterized in that The device also includes: A cleaning monitoring unit is used to obtain monitoring indicator data of each data operation node and display each monitoring indicator data; the monitoring indicator data includes monitoring indicators of multiple cleaning strategies and corresponding indicator values; the indicator values ​​are collected in real time based on the data operation node calling the preset core class.

8. The device according to claim 7, characterized in that The cleaning monitoring unit is also used for: The monitoring index data of each data operation node when the cleaning task is completed are merged to obtain unified index data of multiple cleaning strategies when the cleaning task is completed; the unified index data includes the monitoring index of the cleaning strategy and the corresponding effective index value; the effective index value is the sum of the index values ​​shown in each monitoring index data.

9. A storage medium, characterized in that: The storage medium includes a stored program, wherein the program, when executed by a processor, executes the large model-oriented data cleaning method described in any one of claims 1 to 4.

10. An electronic device, characterized in that: include: processor, memory, and bus; The processor is connected to the memory via the bus; The memory is used to store programs, and the processor is used to run programs, wherein the program, when run by the processor, executes the large model-oriented data cleaning method described in any one of claims 1-4.

Citation Information

Patent Citations

  • Traffic big data cleaning method based on Hadoop and Spark framework and related device

    CN110502509A

  • Big data cleaning method and device, computer equipment and storage medium

    CN113836131A

  • Data cleaning method and device, computer equipment and storage medium

    CN117407384A

  • Method and system for converting structured data into text data set based on task scheduling

    CN118503233A

  • Data processing method and apparatus, electronic device, and storage medium

    WO2022083197A1