A data processing method, a computer readable storage medium and an electronic device

Abstract

The application discloses a data processing method, a computer readable storage medium and an electronic device. The method comprises the following steps: obtaining to-be-cleaned data and an attribute identifier of the to-be-cleaned data, wherein the attribute identifier is used for indicating the source and / or format of the to-be-cleaned data; determining at least one cleaning rule of the to-be-cleaned data according to the attribute identifier and the requirement of a training task; cleaning the to-be-cleaned data according to the at least one cleaning rule to obtain cleaned result data; and storing the cleaned result data in a first storage path corresponding to the attribute identifier and the requirement of the training task. According to the scheme, massive data with different sources and formats can be quickly and efficiently cleaned.

Description

Technical Field

[0001] This application relates to the field of big data processing technology, and in particular to a data processing method, a computer-readable storage medium, and an electronic device. Background Technology

[0002] Currently, the model training process for machine learning algorithms mainly includes the following steps: obtaining training samples; setting labels for the training samples according to the training objectives; inputting the training samples and labels into the selected machine learning algorithm for training; and obtaining the final model after iterative training with a large number of training samples.

[0003] Because the amount of training sample data is very large, it is inevitable that it must be obtained through multiple channels. For example, when training on speech signals, the sample data related to the speech signals may come from different video websites, different films and television shows, or user-recorded audio; when training on images, the sample data related to the images may come from different stock photo websites or user-taken photos. Therefore, sample data obtained directly from different sources may have different formats.

[0004] During model training, the input training samples have certain requirements. Therefore, data cleaning is necessary to obtain training samples that meet the training requirements. However, different data cleaning rules cannot be applied to sample data of different formats. For example, data cleaning rules for text format require setting global filtering conditions and data dependency conditions, while data cleaning rules for audio or image formats do not require these settings. This results in relatively low data cleaning efficiency when dealing with massive amounts of sample data. Summary of the Invention

[0005] This application aims to address the technical problem of low cleaning efficiency in the prior art when cleaning large amounts of sample data. To this end, this application proposes a data processing method, a computer-readable storage medium, and an electronic device.

[0006] To address the aforementioned technical problems, this application provides the following technical solution:

[0007] In a first aspect, some embodiments of this application provide a data processing method, including: obtaining data to be cleaned and an attribute identifier of the data to be cleaned, the attribute identifier being used to indicate the source and / or format of the data to be cleaned; determining at least one cleaning rule for the data to be cleaned based on the attribute identifier and the requirements of the training task; cleaning the data to be cleaned according to the at least one cleaning rule to obtain cleaned result data; and storing the cleaned result data in a first storage path corresponding to the attribute identifier and the requirements of the training task.

[0008] In some embodiments of the data processing method, cleaning the data to be cleaned according to at least one cleaning rule to obtain the cleaning result data includes: when the cleaning rules include multiple cleaning rules, sorting the cleaning rules according to the attribute identifier and the requirements of the training task; according to the sorting, cleaning the data to be cleaned according to multiple cleaning rules in sequence, wherein the data obtained after the previous cleaning rule is cleaned is used as the data to be cleaned in the current cleaning rule, and the data obtained after the last cleaning rule is cleaned is used as the cleaning result data.

[0009] In some embodiments of the data processing method, before obtaining the data to be cleaned and the attribute identifier of the data to be cleaned, the method further includes: obtaining raw data from at least one source; dividing the raw data into data blocks of a set size corresponding to the format according to the format of the raw data; and using all the data blocks as the data to be cleaned.

[0010] In some embodiments of the data processing method, dividing the original data into data blocks of a set size corresponding to the format of the original data includes: storing the data blocks to a second storage path corresponding to the source of the data blocks; wherein data blocks from different sources correspond to different second storage paths.

[0011] In some embodiments of the data processing method, obtaining the data to be cleaned and the attribute identifier of the data to be cleaned includes: obtaining the data block stored under each second storage path, and storing the data block under the third storage path corresponding to the attribute identifier.

[0012] In some embodiments of the data processing method, obtaining the data block stored in each second storage path and storing the data block in a third storage path corresponding to the attribute identifier includes: storing the data block in each second storage path in a corresponding third storage path through multiple parallel download processes.

[0013] In some embodiments, the data processing method stores data blocks from each second storage path to a corresponding third storage path through multiple parallel download processes. This includes: obtaining the number of data blocks stored in each second storage path; obtaining the number of download processes used to retrieve data blocks from each second storage path; determining the data blocks that each download process needs to download based on the number of data blocks and the number of parallel download processes; and starting multiple parallel download processes, with each download process downloading its required data blocks to the corresponding third storage path. Each download process downloads at least one data block, and different download processes download data blocks in parallel.

[0014] In some embodiments of the data processing method, cleaning the data to be cleaned according to at least one cleaning rule to obtain cleaning result data includes: in response to any data block being downloaded to a third storage path, cleaning the downloaded data block in the third storage path according to at least one cleaning rule and obtaining cleaning result data.

[0015] In some embodiments of the data processing method, in response to any data block being downloaded to the third storage path, before cleaning the downloaded data block in the third storage path according to at least one cleaning rule and obtaining the cleaning result data, the method further includes: configuring at least one more cleaning process for each cleaning rule according to the number of download processes, so that the cleaning process corresponds one-to-one with the download process.

[0016] In some embodiments, the data processing method further includes: aggregating the cleaned result data and storing it in a fourth storage path corresponding to the requirements of attribute identification and training tasks.

[0017] In some embodiments, the data processing method further includes uploading the cleaned result data and / or the aggregated cleaned result data to a fifth storage path corresponding to the requirements of attribute identification and training tasks.

[0018] Secondly, some embodiments of this application also provide a data processing apparatus, including: a data acquisition module for acquiring data to be cleaned and attribute identifiers of the data to be cleaned, wherein the attribute identifiers are used to indicate the source and / or format of the data to be cleaned; a cleaning rule determination module for determining at least one cleaning rule for the data to be cleaned based on the attribute identifiers and the requirements of the training task; a cleaning module for cleaning the data to be cleaned according to at least one cleaning rule to obtain cleaning result data; and a storage module for storing the cleaning result data in a first storage path corresponding to the attribute identifiers and the requirements of the training task.

[0019] Thirdly, some embodiments of this application also provide a computer-readable storage medium storing program code executable by a device, the program code including a data processing method for performing any of the first aspects above.

[0020] Fourthly, some embodiments of this application also provide an electronic device, which includes at least one processor and at least one memory, wherein the at least one memory stores program instructions, and the at least one processor is used to invoke the program instructions to perform a data processing method as described in any of the first aspects above.

[0021] The technical solution of this application has the following technical advantages over the prior art:

[0022] The data processing method, computer-readable storage medium, and electronic device provided in this application, after obtaining the data to be cleaned, determine at least one cleaning rule for the data to be cleaned based on the attribute identifier of the data to be cleaned and the requirements of the training task, and directly use the determined cleaning rule to clean the data to be cleaned, thereby enabling the cleaned data to meet the requirements of the data source and / or format, and also to meet the requirements of the training task, ensuring that the machine learning algorithm can quickly and efficiently complete the cleaning of massive amounts of data from different sources and in different formats during training. Attached Figure Description

[0023] The preferred embodiments of this application will be described in detail below with reference to the accompanying drawings, which will help to understand the purpose and advantages of this application, wherein:

[0024] Figure 1 This is a flowchart of a data processing method according to one embodiment of this application;

[0025] Figure 2 This is a schematic diagram illustrating the process of cleaning the data to be cleaned according to a three-stage cleaning rule, as described in one embodiment of this application.

[0026] Figure 3 This is a schematic diagram of the data acquisition process according to one embodiment of this application;

[0027] Figure 4 This is a schematic diagram illustrating the process of acquiring the data to be cleaned and performing three cleaning operations according to one embodiment of this application;

[0028] Figure 5 This is a schematic diagram illustrating the execution of the download and cleaning processes of a data block according to an embodiment of this application;

[0029] Figure 6 This is a structural block diagram of a data processing apparatus according to one embodiment of this application;

[0030] Figure 7 This is a schematic diagram of the hardware connection relationship of an electronic device according to one embodiment of this application. Detailed Implementation

[0031] The technical solutions of this application will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0032] In the description of this application, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this application. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0033] In the description of this application, it should be noted that, unless otherwise expressly specified and limited, the terms "installation," "connection," and "joining" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal communication between two components. Those skilled in the art can understand the specific meaning of the above terms in this application based on the specific circumstances.

[0034] Furthermore, the technical features involved in the different embodiments of this application described below can be combined with each other as long as they do not conflict with each other.

[0035] The following solution provided in this application offers a data processing method for cleaning sample data from multiple sources and in various formats. Furthermore, the cleaning rules can be determined based on the attribute identifiers of the sample data and the requirements of the training task. Therefore, once the attribute identifiers and requirements of the sample data are determined, data cleaning can be completed quickly. This solves the problem of low efficiency in cleaning data from different sources and in different formats in existing technologies.

[0036] This embodiment provides a data processing method that can be applied to the master node of a cluster system performing data cleaning, such as... Figure 1 As shown, the method may include the following steps:

[0037] S101: Obtain the data to be cleaned and its attribute identifiers. The attribute identifiers indicate the source and / or format of the data. The data to be cleaned can have different sources, such as different websites or user-generated content. The data formats are diverse, including image data formats like JPEG, TIFF, and RAW; video data formats like MPG, DAT, and MP4; audio data formats like WMA, MP3, and MPEG; and text data formats like ASCII, MIME, and TXT. In some cases, if the source is determined, the format may also be determined; in this case, including the source in the attribute identifier is sufficient. In other cases, if the source of the data to be cleaned may not have a significant impact on data cleaning, the attribute identifier can simply include the format. Generally, the sources of the data to be cleaned are diverse, and the formats are also varied. In such cases, the attribute identifier includes both the source and the format, and the data cleaning process is designed from these two dimensions.

[0038] S102: Determine at least one cleaning rule for the data to be cleaned based on the attribute identifiers and the requirements of the training task. The requirements of the training task can be determined by the user based on the actual scenario in which their model is trained. Taking speech data as an example, there are usually many different cleaning rules for the source and format of speech data. Among these many rules, the cleaning rules for the speech data are further selected according to different training tasks. For example, judging the age of the speaker in the speech data or judging the meaning expressed by the speech data will result in different cleaning rules for these two training tasks. The former training task may not care about the clarity of keywords in the speech data, while the latter requires the ability to identify clear keyword information. In actual implementation, the user can confirm the cleaning rules according to the actual needs.

[0039] S103: Clean the data to be cleaned according to at least one cleaning rule to obtain the cleaned result data. Once the cleaning rules are determined, the data cleaning operation can be performed according to the cleaning rules, and the final cleaned result data can meet the requirements of the training task.

[0040] S104: Store the cleaning result data in a first storage path corresponding to the attribute identifier and training task requirements. This first storage path can be represented as a logical address and can be specified by the user or predefined and stored. When the user needs to perform a training task, they can call the cleaning result data, for example, as training sample data.

[0041] The above embodiments, after obtaining the data to be cleaned, determine at least one cleaning rule for the data to be cleaned based on the attribute identifier of the data to be cleaned and the requirements of the training task. The determined cleaning rule is then used to clean the data to be cleaned, thereby enabling the cleaned data to meet the requirements of the data source and / or format, as well as the requirements of the training task. This satisfies the need for machine learning algorithms to clean massive amounts of data quickly and efficiently during training.

[0042] As previously stated, the data processing method provided in this embodiment considers both the source and format of the data to be cleaned, as well as the requirements of the training task, when determining the cleaning rules. When multiple cleaning rules are included, they are ordered according to attribute identifiers and training task requirements; the data to be cleaned is then cleaned sequentially according to the order of the multiple cleaning rules, with the data cleaned by the previous cleaning rule serving as the data to be cleaned for the current cleaning rule. Figure 2 Taking the data processing flow shown as an example, it includes data to be cleaned 100 and three cleaning rules: first cleaning rule 201, second cleaning rule 202, and third cleaning rule 203. First cleaning rule 201 obtains the first data to be cleaned, S1 (assuming it's in text format), from the data to be cleaned 100. After cleaning, it obtains the first cleaned data, S2 (for example, the first cleaning rule converts all files with different encoding formats into files with a unified encoding format, such as rewriting TXT, CSV, etc., files into JSON files with the same schema; the first cleaned data, S2, is then a file with a unified format). The first cleaned data, S2, can be stored in the same or different storage space as the data to be cleaned 100. Second cleaning rule 202 obtains the first cleaned data, S2, and after cleaning, it obtains the second cleaned data, S3 (for example...). The same filtering rules are applied to each file to filter out files that do not meet the rules, and the files are rewritten to new files (i.e., the second cleaned data S3 all meet the same filtering rules). The second cleaned data S3 can be stored in the same or different storage space as the data to be cleaned 100. The third cleaning rule 203 obtains the second cleaned data S3 and performs the cleaning to obtain the third cleaned data S4 (for example, reading each line of data in each file and performing global deduplication on the keywords. The deduplication method can use Redis transactions to determine whether the keyword to be deduplicated exists. If it exists, the data is filtered out, i.e., there are no duplicate keywords in the text of the third cleaned data S4). The third cleaned data S4, as the cleaning result data, can be stored in the same or different storage space as the data to be cleaned 100.

[0043] With this solution, the selection, number, and order of cleaning rules can all be configured, thus enabling this application to have a wider range of applications.

[0044] In some embodiments, the data processing methods provided, such as Figure 3 and Figure 4 As shown, the following may be included before step S101:

[0045] S1001: Obtain raw data from at least one source. Sources can be different websites on the internet, movies, digital records, etc. Figure 4 As shown, the source AD ​​is used as an illustration, but the actual sources are not limited to four. Raw data refers to data obtained directly from the source.

[0046] S1002: Divide the original data into data blocks of a set size corresponding to the format of the original data; use all data blocks S0 as the subsequent data to be cleaned 100. The set size of the data blocks can be determined based on the information content. Generally, text data of the same size contains more information, while video data contains less. When dividing the data into blocks, the difference in information content between data blocks of different formats should be minimized. At the same time, to facilitate the application of this method to the processing of massive amounts of data while maintaining high efficiency, the size of the data blocks should not be too small. For example, the data block size for text format can be 100G, the data block size for audio format can be 1T, and the data block size for video format can be 10T. Of course, the set size can be selected based on factors such as the amount of data to be cleaned and the processing efficiency of the system, and is not limited to the examples given above.

[0047] In this solution, the data to be cleaned is cleaned in the form of data blocks, and the size of the data blocks is determined according to the format. This facilitates the retrieval and storage of the data to be cleaned in subsequent processing, making the data processing process more convenient and further improving the efficiency of data cleaning.

[0048] Further, step S1002 includes: storing the data block to a second storage path 101 corresponding to its source, whereby the second storage path 101 can use the same representation as the first storage path, i.e., represented in the form of a logical address; wherein data blocks from different sources correspond to different second storage paths 101. It should be noted that raw data from the same source may contain raw data in different formats. In this case, the format of the data blocks can be distinguished according to their size, naming, etc., meaning that the number and format of data blocks obtained from each source are determined based on their actual situation. In this scheme, data blocks from the same source are stored under the same storage path, while the storage paths for different sources are different. This facilitates the storage of raw data and improves the convenience of subsequent data processing.

[0049] In some preferred embodiments, step S101 may include: retrieving data blocks stored under each second storage path 101 and storing the data blocks under the corresponding third storage path. That is, data blocks of the data to be cleaned from each source can be temporarily stored under the second storage path and then transferred to the third storage path. This allows the second storage path to be released promptly, enabling continuous retrieval and storage of raw data from the source. Furthermore, to improve the efficiency of data cleaning, during the transfer of data blocks from the second storage path to the third storage path, multiple parallel download processes can be used to store the data blocks from each second storage path to the corresponding third storage path. Parallel downloading of data blocks offers higher download efficiency. Specifically, this can be performed as follows:

[0050] S1011: Obtain the number of data blocks stored under each second storage path. As mentioned before, the format and number of data blocks stored under the second storage paths corresponding to different sources are not the same.

[0051] S1012: Obtain the number of download processes used to retrieve data blocks from each second storage path. For a computer, transferring data from a temporary second storage path to a third storage path is equivalent to performing a data download operation. The number of parallel download processes here can be selected based on the system's processing power and the size of the data blocks.

[0052] S1013: Determine the number of data blocks to be downloaded for each download process based on the number of data blocks and the number of download processes;

[0053] S1014: Start multiple download processes in parallel, so that each download process downloads the data blocks it needs to download to the corresponding third storage path. Each download process downloads at least one data block, and different download processes download data blocks in parallel.

[0054] In the implementation, each download process and data block can be assigned a different sequence number. The sequence number of the download process corresponds to the sequence number of the data block, which facilitates allocation. After being assigned a data block to download, each download process will download the data in the data block to the corresponding storage path according to the format. Each download process can download one or more data blocks. For example, download process with sequence number 1 is configured to download three data blocks with sequence numbers 0, 49, and 99; download process with sequence number 2 is configured to download three data blocks with sequence numbers 1, 50, and 100; download process with sequence number 3 is configured to download two data blocks with sequence numbers 2 and 51, and so on.

[0055] When rapidly downloading data blocks to the third storage path via a parallel download process, to improve the efficiency of data cleaning, step S103 can be implemented as follows: In response to any data block being downloaded to the third storage path, the downloaded data blocks in the third storage path are cleaned according to at least one cleaning rule, and the cleaned result data is obtained. That is, for any data block, the download process and the cleaning process are executed in a serial processing manner, and any data block can be cleaned immediately after downloading. More preferably, as... Figure 5 As shown, in some schemes, step S103 involves configuring at least one cleaning process for each cleaning rule based on the download process, ensuring a one-to-one correspondence between cleaning processes and download processes. Once any download process has completely downloaded its required data block to the corresponding third storage path, the cleaning process corresponding to the cleaning rule cleans the completely downloaded data block and obtains the cleaned result data. That is, during data cleaning, multiple parallel cleaning processes are used to clean the data blocks to improve data cleaning efficiency. Figure 5As can be seen, for the same data block, the steps of downloading the data block from the second storage path to the third storage path and the steps of cleaning the data block are processed serially, with a sequential relationship. The data block must be downloaded first before cleaning is performed. However, the multiple processes performing the download step and the multiple processes performing the cleaning step are executed in parallel to achieve higher data processing efficiency. Assuming there are 150 data blocks in the second storage path, with sequence numbers 1-150, and M download and cleaning processes (M=50 in this scheme), each download and cleaning process needs to process 3 data blocks. For example, download process 1 processes data blocks 1, 51, and 101; download process 2 processes data blocks 2, 52, and 102; and so on, with download process 50 processing data blocks 50, 100, and 150. Download process 1 downloads data blocks 1, 51, and 101 sequentially. After data block 1 is downloaded, cleaning process 1 immediately begins cleaning it. Similarly, after data block 51 is downloaded, cleaning process 1 begins cleaning it, and so on, until cleaning process 1 for the last cleaning rule completes its cleaning. Clearly, while cleaning process 1 is cleaning data block 1, download process 1 may be downloading data block 51. Data blocks 1-50 are theoretically downloaded in parallel by download processes 1-50 simultaneously. In this solution, by setting up multi-threaded parallel processing in the above process, the data transfer and data cleaning are carried out in a multi-threaded manner, which can greatly improve the efficiency of data cleaning.

[0056] The data processing methods provided in the above solutions mainly involve the acquisition and cleaning of the data to be cleaned. In practical applications, data aggregation and data uploading steps can also be set according to user needs. That is, the above methods can also include the following steps:

[0057] S105: Aggregate the cleaning results data and store them in the fourth storage path corresponding to the requirements of attribute identification and training tasks.

[0058] S106: Upload the cleaning result data and / or the aggregated cleaning result data to the fifth storage path corresponding to the requirements of attribute identification and training tasks.

[0059] When aggregating data, existing aggregation functions can be selected. The appropriate aggregation function should be chosen based on the format of the cleaned data. As mentioned earlier, when cleaning the data to be cleaned, the source and format were considered. Therefore, when aggregating the cleaned data, the purpose is also to associate different cleaned data under the first storage path based on their source and format. The aggregated data can be stored separately under the fourth transmission path. Assuming step S106 exists, it facilitates the upload step. The data uploaded in step S106 can be the cleaned data from step S104, the aggregated cleaned data from step S105, or this data upload step can be omitted. Steps S105 and S106 can be configured according to user needs.

[0060] like Figure 6 As shown, this embodiment provides a data processing apparatus, including:

[0061] The data acquisition module 61 acquires the data to be cleaned and the attribute identifier of the data to be cleaned. The attribute identifier is used to indicate the source and / or format of the data to be cleaned.

[0062] The cleaning rule determination module 62 determines at least one cleaning rule for the data to be cleaned based on the attribute identifier and the requirements of the training task.

[0063] The cleaning module 63 cleans the data to be cleaned according to at least one cleaning rule to obtain the cleaning result data.

[0064] Storage module 64 stores the cleaning result data to a first storage path corresponding to the attribute identifier and training task requirements.

[0065] The data processing device described above, after acquiring the data to be cleaned, determines at least one cleaning rule for the data to be cleaned based on the attribute identifiers of the data to be cleaned and the requirements of the training task. It then directly uses the determined cleaning rule to clean the data to be cleaned, thereby enabling the cleaned data to meet the requirements of the data source and / or format, as well as the requirements of the training task. This satisfies the need for machine learning algorithms to clean massive amounts of data quickly and efficiently during training.

[0066] Preferably, the data processing apparatus further includes a raw data processing module, which acquires raw data from at least one source, divides the raw data into data blocks of a set size corresponding to the format, and uses all data blocks as the data to be cleaned later. The source can be different websites on the internet, movies, electronic records, etc., and the raw data is the data directly requested from the source. In this scheme, the size of the data block is determined according to the format, which facilitates the retrieval and storage of the data to be cleaned in subsequent processing, making the data processing process more convenient and further improving the efficiency of data cleaning. In the above scheme, the raw data processing module is also used to store the data blocks to a second storage path corresponding to the source. In this scheme, data blocks from the same source are stored in the same storage path, while data blocks from different sources are stored in different storage paths, thus facilitating the storage of raw data and improving the convenience of subsequent data processing.

[0067] In the above scheme, the data acquisition module 61 retrieves data blocks stored in each of the second storage paths and stores the data blocks in the third storage path corresponding to the attribute identifier. Data blocks of the data to be cleaned from each source can be temporarily stored in the second storage path first, and then transferred to the third storage path. This allows the second storage path to be released promptly, enabling continuous retrieval and storage of raw data from the source. Specifically, the data acquisition module 61 uses multiple parallel download processes to store data blocks from each second storage path to the corresponding third storage path, thereby improving the data block transfer rate.

[0068] Further, the data acquisition module 61 acquires the number of data blocks stored under each second storage path; acquires the number of download processes used to acquire data blocks from each second storage path; determines the data blocks that each download process needs to download based on the number of data blocks and the number of download processes; and starts multiple download processes, so that each download process downloads its required data blocks to the corresponding third storage path, wherein each download process downloads at least one data block, and different download processes download data blocks in parallel. Parallel downloading improves data download efficiency. Correspondingly, the cleaning module 63, in response to any data block being downloaded to the third storage path, cleans the downloaded data blocks under the third storage path according to at least one cleaning rule and obtains the cleaned result data. That is, for any data block, its download process and cleaning process are executed serially, and any data block can be cleaned immediately after downloading. Specifically, the cleaning module 63 configures at least one cleaning process for each cleaning rule according to the download process, so that the cleaning process corresponds one-to-one with the download process. Once any download process has completely downloaded its required data block to the corresponding third storage path, the corresponding cleaning process in the cleaning rules cleans the completely downloaded data block and obtains the cleaned result data. By setting up multi-threaded parallel processing in the above process, both data transfer and data cleaning can be performed in a multi-threaded manner, which can greatly improve the efficiency of data cleaning.

[0069] Furthermore, the above data processing device also includes a data aggregation module and / or a data upload module. The data aggregation module aggregates the cleaned result data and stores it in a fourth storage path corresponding to the requirements of attribute identification and training tasks. The data upload module uploads the cleaned result data and / or the aggregated cleaned result data to a fifth storage path corresponding to the requirements of attribute identification and training tasks. These modules can be configured according to user needs, enabling the data processing device to meet a wider range of requirements.

[0070] In some embodiments of this application, a computer-readable storage medium is also provided, which stores program code executable by a device, the program code including data processing methods provided by the schemes in any of the above method embodiments.

[0071] Some embodiments of this application also provide an electronic device, such as... Figure 7As shown, the electronic device includes at least one processor 71 and at least one memory 72. The at least one memory 72 stores program instructions, and the at least one processor 71 is used to invoke the program instructions to execute the data processing method of any of the above-described method embodiments. The electronic device may further include an input device 73 and an output device 74. The processor 71, memory 72, input device 73, and output device 74 can be communicatively connected. The memory 72, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules. The processor 71 executes various functional applications and data processing by running the non-volatile software programs, instructions, and modules stored in the memory 72, thereby implementing the data processing method provided by any of the above-described embodiments. This device has high data cleaning efficiency, and the cleaning rules, data aggregation, and data uploading steps can be configured as needed, exhibiting excellent scalability.

[0072] Obviously, the above embodiments are merely illustrative examples for clear explanation and are not intended to limit the implementation. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is neither necessary nor possible to exhaustively list all possible implementations here. However, obvious variations or modifications derived therefrom are still within the scope of protection of this application.

Claims

1. A data processing method, characterized in that, include: Obtain raw data from at least one source; The original data is divided into data blocks of a set size corresponding to the format of the original data, and all data blocks are used as data to be cleaned; the difference in the amount of information contained in data blocks of different formats is within a preset range; when the original data from the same source contains original data of different formats, the format of the data blocks is distinguished according to the size of the data blocks. The data block is stored in a second storage path corresponding to its source; wherein data blocks from different sources correspond to different second storage paths. Obtain the data to be cleaned and the attribute identifier of the data to be cleaned, wherein the attribute identifier is used to indicate the source and format of the data to be cleaned, including: storing each data block under the second storage path to the corresponding third storage path through multiple parallel download processes; At least one cleaning rule is determined for the data to be cleaned based on the attribute identifiers and the requirements of the training task; the requirements of the training task are determined based on the scenario in which the model training is applied. The data to be cleaned is cleaned according to at least one cleaning rule to obtain cleaned result data; including: in response to any data block being downloaded to a third storage path, the downloaded data block under the third storage path is cleaned according to at least one cleaning rule and cleaned result data is obtained; when performing data cleaning, the data block is cleaned through multiple parallel cleaning processes; The cleaning result data is stored in a first storage path corresponding to the attribute identifier and the requirements of the training task.

2. The data processing method according to claim 1, characterized in that, The step of cleaning the data to be cleaned according to the at least one cleaning rule to obtain the cleaned result data includes: When the at least one cleaning rule includes multiple cleaning rules, the cleaning rules are sorted according to the attribute identifier and the requirements of the training task; According to the sorting, the data to be cleaned is cleaned sequentially according to the multiple cleaning rules. The data obtained after the previous cleaning rule is cleaned is used as the data to be cleaned in the current cleaning rule, and the data obtained after the last cleaning rule is cleaned is used as the cleaning result data.

3. The data processing method according to claim 1, characterized in that, The step of storing each data block under the second storage path to the corresponding third storage path through multiple parallel download processes includes: Get the number of data blocks stored under each second storage path; Get the number of download processes used to retrieve data blocks from each second storage path; Based on the number of data blocks and the number of download processes, determine the data blocks that each download process needs to download; The multiple download processes are started, so that each download process downloads the data blocks it needs to download to the corresponding third storage path. Each download process downloads at least one data block, and different download processes download data blocks in parallel.

4. The data processing method according to claim 1, characterized in that, Before cleaning the downloaded data blocks in the third storage path according to the at least one cleaning rule and obtaining the cleaning result data in response to any data block being downloaded to the third storage path, the method further includes: Configure at least one cleaning process for each cleaning rule based on the number of download processes, so that there is a one-to-one correspondence between the cleaning process and the download process.

5. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores program code executable by the device, the program code including methods for performing data processing as described in any one of claims 1-4.

6. An electronic device, characterized in that, The electronic device includes at least one processor and at least one memory, wherein the at least one memory stores program instructions, and the at least one processor is used to invoke the program instructions to execute the data processing method as described in any one of claims 1-4.

Images