Data file processing method and device, equipment, storage medium and program product

Abstract

The invention discloses a data file processing method and device, equipment, a storage medium and a program product, and relates to the technical field of big data processing, and the data file processing method comprises the following steps: reading a plurality of to-be-merged data files; performing file merging on the plurality of to-be-merged data files through a preset clustering table service function to obtain a merged metadata file; and offline cleaning is performed on a data file corresponding to the merged metadata file through a preset offline cleaning function, and the offline cleaning function is independently deployed for the clustering table service function. According to the method and the device, the file merging of the data lake clustering table service and the independently deployed file offline cleaning are realized, the problem that the clustering function cannot be stably executed due to the coupling between the data lake data file merging and cleaning in the related technology is solved, and the memory occupation caused by the serial execution of the file cleaning of the data lake clustering table service is avoided; and the execution stability of the data file clustering function is improved.

Description

Technical Field

[0001] The present application relates to the field of big data processing technology, and in particular to a data file processing method, device, equipment, storage medium and program product. Background Art

[0002] In the big data scenario, the data lake, as a storage place for the entire data set, will contain a large number of data files. Therefore, a clustering table service function is proposed that can merge small files to optimize file layout. The main workflow of the clustering table service function is to read small files, merge and sort, output large files, and clean up old files.

[0003] At present, the relevant technology uses the serial execution of file merging and cleaning to perform the clustering function, coupling the file merging and cleaning processes, which will store a large amount of metadata generated by the file merging in the memory. The deletion of the merged files takes a long time, resulting in the inability to release the memory in time, which in turn causes the failure of the entire clustering function to execute.

[0004] The above contents are only used to assist in understanding the technical solution of the present application and do not constitute an admission that the above contents are prior art. Summary of the invention

[0005] The main purpose of this application is to provide a data file processing method, device, equipment, storage medium and program product, aiming to solve the technical problem that the clustering function cannot be stably executed due to the coupling between the merging and cleaning of data files in the data lake in the related technology.

[0006] To achieve the above objectives, the present application proposes a data file processing method, which is applied to a data lake and includes:

[0007] Read several data files to be merged;

[0008] Merging the plurality of data files to be merged through a preset clustering table service function to obtain a merged metadata file;

[0009] The data files corresponding to the merged metadata files are cleaned offline through a preset offline cleaning function, and the offline cleaning function is independently deployed for the clustering table service function.

[0010] In one embodiment, before the step of merging the plurality of data files to be merged by using a preset clustering table service function to obtain a merged metadata file, the step further includes:

[0011] Get the initial clustering table service function;

[0012] Based on the parent class abstract rich function of the preset asynchronous cleanup function, the initial clustering table service function is functionally rewritten to obtain the clustering table service function.

[0013] In one embodiment, the initial cluster table service function includes an initial cluster submission output operator, and the initial cluster submission output operator is obtained by inheriting the asynchronous cleanup function.

[0014] The step of rewriting the initial clustering table service function based on the parent class abstract rich function of the preset asynchronous cleanup function to obtain the clustering table service function includes:

[0015] Reinheriting the parent class abstract rich function of the asynchronous cleanup function through the initial cluster submission output operator to obtain the re-inherited cluster submission output operator;

[0016] Based on the re-inherited cluster submission output operator, the function function interface corresponding to the initial cluster table service function is functionally rewritten to obtain the cluster table service function.

[0017] In one embodiment, the step of merging the plurality of data files to be merged to obtain a merged metadata file by using a preset clustering table service function includes:

[0018] Merging the plurality of data files to be merged to obtain a target merged data file;

[0019] Through the cluster submission output operator of the cluster table service function, information statistics are performed on the plurality of data files to be merged and the target merged data file to generate a merged metadata file.

[0020] In one embodiment, the step of performing offline cleaning of the data files corresponding to the merged metadata files by using a preset offline cleaning function includes:

[0021] Reading metadata information in the merged metadata file, the metadata information at least including a file name, a file status, and a file path;

[0022] Creating an offline scheduled task according to the metadata information;

[0023] When the system time meets the preset scheduling period of the offline scheduled scheduling task, the data file corresponding to the merged metadata file is cleaned up offline through the offline scheduled scheduling task.

[0024] In one embodiment, the step of performing offline cleaning of the data files corresponding to the merged metadata files through the offline scheduled task includes:

[0025] Parsing the metadata information of the merged metadata file to obtain file information of the data file to be cleaned up;

[0026] The interface of the file system is called, and the corresponding data files to be cleaned in the file system are cleaned offline according to the file information through the interface of the file system.

[0027] In addition, to achieve the above-mentioned purpose, the present application also proposes a data file processing device, the data file processing device comprising:

[0028] A file reading module, used to read several data files to be merged;

[0029] A file merging module, used to merge the plurality of data files to be merged through a preset clustering table service function to obtain a merged metadata file;

[0030] The offline cleaning module is used to perform offline cleaning on the data files corresponding to the merged metadata file through a preset offline cleaning function, wherein the offline cleaning function is independently deployed for the clustering table service function.

[0031] In addition, to achieve the above-mentioned purpose, the present application also proposes a data file processing device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, and the computer program is configured to implement the steps of the data file processing method described above.

[0032] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the data file processing method described above are implemented.

[0033] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, the steps of the data file processing method described above are implemented.

[0034] The present application provides a data file processing method, which first reads multiple data files that need to be merged to determine the objects of file merging; merges the multiple data files that need to be merged by using a preset clustering table service function to obtain a merged metadata file, so that the clustering table service function performs a single file merging operation; and uses an offline cleaning function that is independently deployed in advance for the clustering table service function to delete the data files corresponding to the merged metadata file offline to separate the merging and cleaning processes of data file clustering, thereby avoiding the long-term occupation of memory caused by the metadata required for data file cleaning, thereby improving the stability of the execution of the data file clustering function.

[0035] In summary, this application solves the problem of the coupling between the merging and cleaning of data lake data files in related technologies, which causes the clustering function to be unable to be executed stably, through the file merging of the data lake clustering table service and the offline cleaning of independently deployed files. It avoids the memory occupation caused by the serial execution of file cleaning of the data lake clustering table service and improves the stability of the execution of the data file clustering function. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0037] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0038] Figure 1 A flowchart of the first embodiment of the data file processing method of the present application is provided;

[0039] Figure 2 A flowchart of the second embodiment of the data file processing method of the present application is provided;

[0040] Figure 3 A flowchart of the third embodiment of the data file processing method of the present application is provided;

[0041] Figure 4 This is a schematic diagram of the module structure of the data file processing device according to an embodiment of the present application;

[0042] Figure 5 Schematic diagram of the device structure of the hardware operating environment involved in the data file processing method in the embodiment of the present application.

[0043] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0044] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0045] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0046] The main solution of the embodiment of the present application is: read several data files to be merged; merge the several data files to be merged through a preset clustering table service function to obtain a merged metadata file; and perform offline cleaning of the data files corresponding to the merged metadata file through a preset offline cleaning function, wherein the offline cleaning function is independently deployed for the clustering table service function.

[0047] Technical terms involved in this application:

[0048] Data Lake: It is a centralized repository for storing large-scale structured and unstructured data. Unlike traditional data warehouses, data lakes can store data in raw format, including relational data, semi-structured data and unstructured data, without pre-defining the structure or format of the data. For example, Hudi (Hadoop Upserts Deletes and Incrementals) is a streaming data lake platform.

[0049] Table Service: refers to a managed service provided on a cloud platform for storing structured data and providing functions similar to relational database tables. It is used to manage and query large-scale distributed data sets.

[0050] Clustering: A table service provided by Hudi Data Lake that can merge many small files into a small number of large files, thereby preventing the generation of too many small files. Its main workflow is to read small files, merge, sort, output to large files, and clean up old files.

[0051] Flink: is a distributed stream computing engine used for stateful computing on unbounded and bounded data streams. Flink executes any stream data program in a data-parallel and pipelined manner. Flink's pipeline runtime system can execute batch and stream processing programs.

[0052] In the prior art, the cleanup function and the file merging function provided by the clustering function of the data lake are in the same Flink job, and the cleanup operator occupies one process. In the scenario of large data volume, after the file merging is completed, the file needs to be cleaned up, which requires the metadata of the file to be collected and stored in a Map data structure. If the amount of metadata is too large, a large amount of metadata will be collected and stored in the memory during the process, which will cause the Map data structure to occupy too much memory, and the file deletion after the small files are merged is slow, resulting in the memory occupied by the metadata not being released in time, resulting in a long time for process recycling, resulting in the failure of heartbeat reporting, and then leading to the failure of the execution of the entire Flink job.

[0053] In addition, the clustering function is a serial execution of file merging and cleaning under each partition of the data lake, that is, each partition will not continue to process the next partition until the entire process of "merging and cleaning" is completed. The cleaning operation is not a key factor in determining the overall task completion time, and it takes a long time. It is easy for the entire task to fail due to non-critical path cleaning timeout. If the task fails, it cannot be rolled back to the previous state through the snapshot rollback function, and the clustering task cannot be stably executed.

[0054] This application solves the problem of the coupling between the merging and cleaning of data lake data files in related technologies, which leads to the instability of clustering function execution, by merging files of the data lake clustering table service and cleaning files of independently deployed files offline. It avoids the memory occupation of the serial execution of file cleaning of the data lake clustering table service and improves the stability of the execution of the data file clustering function.

[0055] It should be noted that the execution subject of this embodiment may be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device capable of realizing the above functions, a data file processing device, etc. The following takes a data file processing device as an example to illustrate this embodiment and the following embodiments.

[0056] Based on this, the present application embodiment provides a data file processing method, referring to Figure 1 , Figure 1 This is a flow chart of the first embodiment of the data file processing method of the present application.

[0057] In this embodiment, the data file processing method includes steps S10 to S30:

[0058] Step S10, reading a number of data files to be merged;

[0059] It should be noted that the data lake usually stores raw data files of various formats and sources, such as structured, semi-structured and unstructured data. The data files to be merged can be read from the data lake or an external file system.

[0060] It is understandable that the system will read multiple data files to be merged for further merging and cleaning.

[0061] Step S20, merging the plurality of data files to be merged through a preset clustering table service function to obtain a merged metadata file;

[0062] It should be noted that the clustering table service function aims to integrate smaller data files from multiple sources into one or more larger merged data files by defining and implementing data merge rules. The merged metadata file can be persistently stored in the data lake or in an external file system.

[0063] In addition, it should be noted that the merged metadata file usually contains metadata of the data file, that is, data that describes the data file, and specifically may include information of various data files (data files before the merge and data files after the merge), such as the source, format, timestamp, etc. of the data files; it may also include data of the merge process, such as the path and merge status of the data files before and after the merge.

[0064] It is understandable that the system uses the preset clustering table service function to merge the data files read in the previous step to obtain a merged metadata file for subsequent offline cleaning operations of the data files.

[0065] Step S30: Perform offline cleaning on the data files corresponding to the merged metadata file through a preset offline cleaning function, wherein the offline cleaning function is independently deployed for the clustering table service function.

[0066] It is understandable that since the clustering table service function in the related technology adopts the task partition serial execution of file merging and cleaning, the metadata of the data file cleaning performed by the clustering table service function is avoided to avoid long-term memory occupation affecting the merging of the next partition data file, thereby causing the clustering task execution failure. By using the offline cleaning function independently deployed for the clustering table service function, the data files corresponding to the merged metadata files are cleaned offline, thereby realizing the decoupling of file merging and cleaning functions, avoiding the influence of file cleaning on file merging, and ensuring the stability of clustering task execution.

[0067] In a feasible implementation manner, before the step of merging the plurality of data files to be merged by using a preset clustering table service function to obtain a merged metadata file, the step further includes:

[0068] Step S201, obtaining an initial clustering table service function;

[0069] Step S202: Based on the parent class abstract rich function of the preset asynchronous cleanup function, the initial clustering table service function is rewritten to obtain the clustering table service function.

[0070] It should be noted that the initial clustering table service function refers to the clustering table service function provided by the open source data lake framework, which sets the specific workflow of clustering, including reading small files, merging and sorting, outputting to large files, cleaning up old files, etc. Cleaning up old files is implemented by a preset asynchronous cleaning function.

[0071] It is understandable that in order to customize the clustering table service function to adapt to the independently deployed offline cleaning function, it is first necessary to obtain the initial clustering table service function that includes the basic functions, and then use the parent class abstract rich function of the preset asynchronous cleaning function that implements the asynchronous cleaning operation to rewrite the initial clustering table service function. By rewriting and changing the data file processing logic and metadata management in the initial clustering table service function, a new clustering table service function that meets the data file merging and cleaning requirements is obtained.

[0072] In this implementation, by rewriting and customizing functions, it is ensured that the clustering table service function can meet specific data file merging and offline cleaning requirements, which can effectively integrate system resources for data file merging and cleaning and improve the execution efficiency of clustering tasks.

[0073] In a feasible implementation, the initial cluster table service function includes an initial cluster submission output operator, and the initial cluster submission output operator is obtained by inheriting the asynchronous cleanup function.

[0074] The step of rewriting the initial clustering table service function based on the parent class abstract rich function of the preset asynchronous cleanup function to obtain the clustering table service function includes:

[0075] Step S2021, re-inheriting the parent class abstract rich function of the asynchronous cleanup function through the initial cluster submission output operator to obtain the re-inherited cluster submission output operator;

[0076] It should be noted that the asynchronous cleanup function refers to a function used to asynchronously perform data file cleanup operations. The parent class abstract rich function of the asynchronous cleanup function refers to a base class or interface that provides the basic structure and functions of the cleanup operation for subclasses to extend and implement. The cluster submission output operator refers to an operator that processes and submits the merge results of data files.

[0077] In addition, it should be noted that in object-oriented programming, inheritance allows subclasses to obtain properties and methods from parent classes, and "re-inheritance" means extending or modifying the existing asynchronous cleanup function to support new functions or meet new requirements.

[0078] It can be understood that by re-inheriting the parent class abstract rich function of the asynchronous cleanup function, the initial cluster submission output operator is re-inherited, that is, based on the existing asynchronous cleanup function framework or interface, adaptive modification and expansion are performed to meet the needs and functions of the new cluster submission output operator.

[0079] Step S2022: Based on the re-inherited cluster submission output operator, the function function interface corresponding to the initial cluster table service function is functionally rewritten to obtain the cluster table service function.

[0080] It should be noted that the function interface refers to the interface or method that defines the function call method and function, allowing modules to interact with each other. Function rewriting refers to modifying and re-implementing existing functions to meet new requirements or optimize performance.

[0081] It is understandable that after inheriting the new functional module, the re-inherited cluster submission output operator is functionally rewritten to adapt to the independently deployed offline cleanup function, which specifically involves modifying and optimizing the function interface to ensure that the new cluster submission output operator can process data correctly and be compatible with other parts of the system.

[0082] Exemplarily, the initial cluster submission output operator re-inherits the parent class abstract rich function of the asynchronous cleanup function and can change the asynchronous serial execution of data file merging and cleaning to a non-asynchronous non-serial execution, that is, cancel the inheritance of the asynchronous cleanup function by the cluster submission output operator, and change it to inherit the parent class abstract rich function of the asynchronous cleanup function. After the re-inheritance, when the data file merge of the current partition is completed, it can directly flow to the next partition to continue the data file merge, and generate a timed old data file cleanup task. After all partitions complete the data file merge or a certain time period, or when the system resources are sufficient, the offline cleanup function is called to execute the cleanup task of the old data files and clean up the merged data files.

[0083] In this implementation, through re-inheritance and function rewriting, the system can better adapt to the independently deployed offline cleanup function, ensure that the clustering table service function independently realizes data file merging, avoid the impact of memory usage of data file cleanup on the execution of the entire clustering function, and improve the overall system performance and execution efficiency.

[0084] This embodiment provides a data file processing method, which solves the problem of the coupling between the data lake data file merging and cleaning in the related technology that the clustering function cannot be stably executed, through the file merging of the data lake clustering table service and the offline cleaning of independently deployed files. It avoids the memory occupation caused by the serial execution of the file cleaning of the data lake clustering table service, and improves the stability of the execution of the data file clustering function.

[0085] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the first embodiment can refer to the above introduction, and will not be repeated in the following. Figure 2 , Figure 2The flowchart provided for the second embodiment of the data file processing method of the present application is as follows: the step of merging the plurality of data files to be merged to obtain a merged metadata file by using a preset clustering table service function includes:

[0086] Step S203, merging the plurality of data files to be merged to obtain a target merged data file;

[0087] It should be noted that the operation of merging files can be to merge the contents of files together to form a new file by simple splicing, or to reintegrate the contents of data files by reordering or using an algorithm to form a new file.

[0088] In addition, it should be noted that the data files are pre-processed before merging, such as removing redundant data, format conversion, etc., and then a more complex merging algorithm (such as merging based on data content) is used to ensure that the merged data is more consistent and accurate. In-depth data analysis can be performed on several data files to be merged, such as cluster analysis, multi-dimensional data statistics, etc., to provide more detailed and in-depth data analysis results, which are suitable for complex clustering function requirements.

[0089] It is understandable that there is scattered information that needs to be summarized in several data files to be merged, and the data can be extracted from the data files and integrated into a new target merged data file according to a specific rule or format.

[0090] Step S204: Through the cluster submission output operator of the cluster table service function, information statistics are performed on the plurality of data files to be merged and the target merged data file to generate a merged metadata file.

[0091] It can be understood that after the data files are merged, the merged data is statistically analyzed through the cluster submission output operator of the cluster table service function, and the generated merged metadata file contains detailed statistical information on multiple data files to be merged and the target merged data file, such as the status of the data files before and after the merge, the file path, timestamp, etc.

[0092] In a feasible implementation manner, the step of performing offline cleaning of the data file corresponding to the merged metadata file by using a preset offline cleaning function includes:

[0093] Step 301, reading metadata information in the merged metadata file, the metadata information at least including the file name, file status and file path;

[0094] It should be noted that the file name refers to the file name before and after the data file is cleaned, which can be used as the unique identifier of the data file. The file status refers to the current processing status of the data file, which can be divided into merged, unmerged, cleaned, etc., to determine which data files need to be operated. The file path refers to the storage location of the data file in the data lake or file system. The data file can be found in the corresponding directory through the file path to perform the actual data file merging and cleaning operations.

[0095] It is understandable that in order to obtain information about data files that need to be cleaned up, so as to know which files need to be processed and how to process them, the metadata information stored in the merged metadata file, ie, the key information of the data files, is extracted.

[0096] Step 302, creating an offline scheduled task according to the metadata information;

[0097] It can be understood that an offline scheduled task is created based on the extracted metadata information so as to automatically perform the cleanup operation at a preset time point. The creation of the offline scheduled schedule may include the defined task execution processing logic, frequency (scheduling cycle), and execution parameters (execution time, file path, etc.).

[0098] Step 303: When the system time meets the preset scheduling period of the offline scheduled scheduling task, the data file corresponding to the merged metadata file is cleaned up offline through the offline scheduled scheduling task.

[0099] It is understandable that when the current system time meets the preset scheduling cycle, the actual cleanup operation is performed, including deleting or archiving data files, ensuring that the data files are cleaned up after the merger, thereby releasing storage space and completing the optimized layout of data files.

[0100] In this implementation, by independently deploying an offline cleanup function for the clustering table service function, automatic offline cleanup is achieved after the data files are merged, which reduces the burden on the real-time system, avoids the impact of metadata memory usage of data file cleanup on data file merging, and improves the stability of clustering function execution.

[0101] In a feasible implementation manner, the step of performing offline cleaning of the data files corresponding to the merged metadata files through the offline timed scheduling task includes:

[0102] Step 3031, parsing the metadata information of the merged metadata file to obtain file information of the data file to be cleaned;

[0103] It can be understood that the file content of the merged metadata file is parsed into a data structure (such as a dictionary or a data table), information related to the files to be cleaned is extracted, and necessary information of each data file to be cleaned is obtained, including the file name, file path, file status, etc.

[0104] Step 3032: call the interface of the file system, and perform offline cleaning of the corresponding data files to be cleaned in the file system according to the file information through the interface of the file system.

[0105] It is understandable that the actual data files are operated using the interface provided by the file system, the file system is connected through a programming interface, each data file to be cleaned is located according to the file information, and then the data file is deleted or archived.

[0106] In this implementation, metadata information analysis and offline cleaning are used to ensure the accuracy of the cleaning operation, reduce the risk of accidental file deletion, reduce the impact on the real-time performance of the system, and improve the execution efficiency of the clustering function.

[0107] In this embodiment, the combination of data file merge statistics and automated offline cleaning can adapt to the different complexities and real-time requirements of data file processing, can flexibly handle various data merging situations, facilitate further data file cleaning processing and decision-making, ensure the long-term effectiveness of data file management and the rational use of system resources, reduce the burden on the real-time system, and improve the stability of clustering function execution.

[0108] For example, in order to help understand the implementation process of the data file processing method obtained by combining the first embodiment and the second embodiment, the same or similar contents as the first embodiment and the second embodiment can be referred to the above introduction, and will not be repeated later. Figure 3 , Figure 3 A flowchart diagram of the third embodiment of the data file processing method of the present application.

[0109] Taking the small file merging scenario on Hudi's data lake as an example, data file processing is implemented through Hudi's Clustering table service function, which mainly includes two parts. The first part is the Clustering table service function. The main function of Clustering is to read small files, reorder data, merge them into large files and generate metadata; the second part is the offline Clean function. The main function of offline cleaning is to read metadata and delete old files. Specifically, it includes:

[0110] The first part is the Clustering table service function. The source code of Clustering is modified. The original ClusteringCommitSink operator (i.e. clustering commit output operator) inherits the cleanup function, and re-inherits the parent class abstract rich function of the cleanup function to implement the SinkFunction (output function), CheckpointedFunction (checkpoint function), and CheckpointListener (checkpoint listener) interfaces. The common functions in the parent class cleanup function are mainly rewritten to manage the WriteClient (write client) life cycle. Based on the existing cleanup function, adaptive modifications and extensions are made to meet the scheduling requirements of independently deployed offline cleanup functions.

[0111] After executing Clustering, that is, merging files, in the ClusteringCommitSink operator, when submitting Commit, its original asynchronous cleanup function is turned off, and the single responsibility principle of the Clustering function is followed (that is, the Clustering function only needs to perform the merge operation of data files). The ClusteringCommitSink operator will perform statistical operations on file information, which includes the small files read and the large files generated by merging. Finally, these statistical information will be persisted as metadata to the file system, and the metadata will be stored in the .replacecommit file format. The generation of the metadata file marks the successful execution of the Clustering function.

[0112] The second part is the offline Clean function. The main function of offline Clean is to read the metadata file generated by Clustering in the first part and parse it to obtain the old files that need to be deleted. Finally, the API interface provided by the file system is called to clean up the old files, which not only improves query performance but also reduces storage costs.

[0113] The offline Clean function is a program developed in Java. The main steps of Clean are to first connect to the file system HDFS (Hadoop Distributed File System), read the metadata file of the completed Clustering in the metadata directory of the corresponding Hudi table, parse the metadata file, get the old files to be deleted, and then call the HDFS API to delete the old files. The execution of the offline Clean program is scheduled offline using the scheduling platform, and a reasonable scheduling cycle can be configured according to the number of generated files.

[0114] For the Clustering function, data file cleaning is not the key path for function execution. By deploying the Clean function independently, offline cleaning can be performed on demand (for example, it can be configured to be cleaned after a certain period of time or at night when business is off-peak), and customized timed scheduling can be supported, which greatly improves the execution efficiency of the key path of the clustering function, ensuring the efficient and stable execution of the Clustering function and reducing the coupling of the Clean function to the Clustering function.

[0115] In this embodiment, the file merging of the data lake clustering table service and the offline cleaning of independently deployed files are solved, which solves the problem of the clustering function being unable to be stably executed due to the coupling between the data lake data file merging and cleaning in the related technology, avoids the memory occupation of the serial execution of the file cleaning of the data lake clustering table service, and improves the stability of the execution of the data file clustering function.

[0116] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the data file processing method of the present application. More simple transformations based on this technical concept are all within the scope of protection of the present application.

[0117] This application also provides a data file processing device, please refer to Figure 4 , the data file processing device comprises:

[0118] A file reading module 10, used for reading a number of data files to be merged;

[0119] The file merging module 20 is used to merge the plurality of data files to be merged through a preset clustering table service function to obtain a merged metadata file;

[0120] The offline cleaning module 30 is used to perform offline cleaning on the data files corresponding to the merged metadata file through a preset offline cleaning function, and the offline cleaning function is independently deployed for the clustering table service function.

[0121] Optionally, the file merging module 20 is further used for:

[0122] Get the initial clustering table service function;

[0123] Based on the parent class abstract rich function of the preset asynchronous cleanup function, the initial clustering table service function is functionally rewritten to obtain the clustering table service function.

[0124] Optionally, the initial clustering table service function includes an initial clustering submission output operator, and the initial clustering submission output operator is obtained by inheriting the asynchronous cleanup function. The file merging module 20 is further used for:

[0125] Reinheriting the parent class abstract rich function of the asynchronous cleanup function through the initial cluster submission output operator to obtain the re-inherited cluster submission output operator;

[0126] Based on the re-inherited cluster submission output operator, the function function interface corresponding to the initial cluster table service function is functionally rewritten to obtain the cluster table service function.

[0127] Optionally, the file merging module 20 is further used for:

[0128] Merging the plurality of data files to be merged to obtain a target merged data file;

[0129] Through the cluster submission output operator of the cluster table service function, information statistics are performed on the plurality of data files to be merged and the target merged data file to generate a merged metadata file.

[0130] Optionally, the offline cleaning module 30 is further used for:

[0131] Reading metadata information in the merged metadata file, the metadata information at least including a file name, a file status, and a file path;

[0132] Creating an offline scheduled task according to the metadata information;

[0133] When the system time meets the preset scheduling period of the offline scheduled scheduling task, the data file corresponding to the merged metadata file is cleaned up offline through the offline scheduled scheduling task.

[0134] Optionally, the offline cleaning module 30 is further used for:

[0135] Parsing the metadata information of the merged metadata file to obtain file information of the data file to be cleaned up;

[0136] The interface of the file system is called, and the corresponding data files to be cleaned in the file system are cleaned offline according to the file information through the interface of the file system.

[0137] The data file processing device provided by the present application adopts the data file processing method in the above embodiment, which can solve the technical problem that the coupling between the merging and cleaning of data files in the data lake of the related technology causes the clustering function to be unable to be stably executed. Compared with the prior art, the beneficial effects of the data file processing device provided by the present application are the same as the beneficial effects of the data file processing method provided by the above embodiment, and the other technical features in the data file processing device are the same as the features disclosed in the above embodiment method, which will not be repeated here.

[0138] The present application provides a data file processing device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the data file processing method in the first embodiment described above.

[0139] Reference below Figure 5 , which shows a schematic diagram of the structure of a data file processing device suitable for implementing the embodiment of the present application. The data file processing device in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 5 The data file processing device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0140] like Figure 5As shown, the data file processing device may include a processing device 1001 (e.g., a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the data file processing device are also stored. The processing device 1001, ROM1002, and RAM1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the data file processing device to communicate with other devices wirelessly or by wire to exchange data. Although the data file processing device with various systems is shown in the figure, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or have alternatively.

[0141] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0142] The data file processing device provided by the present application adopts the data file processing method in the above embodiment, which can solve the technical problem that the coupling between the merging and cleaning of data files in the data lake of the related technology causes the clustering function to be unable to be stably executed. Compared with the prior art, the beneficial effects of the data file processing device provided by the present application are the same as the beneficial effects of the data file processing method provided by the above embodiment, and the other technical features in the data file processing device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0143] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0144] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0145] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, and the computer-readable program instructions are used to execute the data file processing method in the above-mentioned embodiment.

[0146] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination of the above.

[0147] The computer-readable storage medium may be included in the data file processing device; or may exist independently without being assembled into the data file processing device.

[0148] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by a data file processing device, the data file processing device: reads a number of data files to be merged; merges the data files to be merged through a preset clustering table service function to obtain a merged metadata file; and performs offline cleaning of data files corresponding to the merged metadata file through a preset offline cleaning function, wherein the offline cleaning function is independently deployed for the clustering table service function.

[0149] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0150] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0151] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0152] The readable storage medium provided by the present application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned data file processing method, and can solve the technical problem that the coupling between the merging and cleaning of data files in the data lake of the related technology causes the clustering function to be unable to be stably executed. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided by the present application are the same as the beneficial effects of the data file processing method provided by the above-mentioned embodiment, and will not be repeated here.

[0153] The present application also provides a computer program product, including a computer program, which implements the steps of the data file processing method as described above when executed by a processor.

[0154] The computer program product provided by this application can solve the technical problem that the coupling between the merging and cleaning of data lake data files in the related technology causes the clustering function to be unable to be stably executed. Compared with the prior art, the beneficial effects of the computer program product provided by this application are the same as the beneficial effects of the data file processing method provided by the above embodiment, which will not be repeated here.

[0155] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A data file processing method, It is characterized in that The method is applied to a data lake, including: Read several data files to be merged; Merging the plurality of data files to be merged through a preset clustering table service function to obtain a merged metadata file; The data files corresponding to the merged metadata files are cleaned offline through a preset offline cleaning function, and the offline cleaning function is independently deployed for the clustering table service function.

2. The method according to claim 1, It is characterized in that Before the step of merging the plurality of data files to be merged by using the preset clustering table service function to obtain a merged metadata file, the step further includes: Get the initial clustering table service function; Based on the parent class abstract rich function of the preset asynchronous cleanup function, the initial clustering table service function is functionally rewritten to obtain the clustering table service function.

3. The method according to claim 2, It is characterized in that The initial cluster table service function includes an initial cluster submission output operator, and the initial cluster submission output operator is obtained by inheriting the asynchronous cleanup function. The step of rewriting the initial clustering table service function based on the parent class abstract rich function of the preset asynchronous cleanup function to obtain the clustering table service function includes: Reinheriting the parent class abstract rich function of the asynchronous cleanup function through the initial cluster submission output operator to obtain the re-inherited cluster submission output operator; Based on the re-inherited cluster submission output operator, the function function interface corresponding to the initial cluster table service function is functionally rewritten to obtain the cluster table service function.

4. The method according to claim 1, It is characterized in that The step of merging the plurality of data files to be merged to obtain a merged metadata file by using a preset clustering table service function includes: Merging the plurality of data files to be merged to obtain a target merged data file; Through the cluster submission output operator of the cluster table service function, information statistics are performed on the plurality of data files to be merged and the target merged data file to generate a merged metadata file.

5. The method according to claim 1, It is characterized in that The step of performing offline cleaning of the data files corresponding to the merged metadata files by using a preset offline cleaning function includes: Reading metadata information in the merged metadata file, the metadata information at least including a file name, a file status, and a file path; Creating an offline scheduled task according to the metadata information; When the system time meets the preset scheduling period of the offline scheduled scheduling task, the data file corresponding to the merged metadata file is cleaned up offline through the offline scheduled scheduling task.

6. The method according to claim 5, It is characterized in that The step of performing offline cleaning of the data files corresponding to the merged metadata files through the offline timed scheduling task includes: Parsing the metadata information of the merged metadata file to obtain file information of the data file to be cleaned up; The interface of the file system is called, and the corresponding data files to be cleaned in the file system are cleaned offline according to the file information through the interface of the file system.

7. A data file processing device, It is characterized in that The device comprises: A file reading module, used to read several data files to be merged; A file merging module, used to merge the plurality of data files to be merged through a preset clustering table service function to obtain a merged metadata file; The offline cleaning module is used to perform offline cleaning on the data files corresponding to the merged metadata file through a preset offline cleaning function, wherein the offline cleaning function is independently deployed for the clustering table service function.

8. A data file processing device, It is characterized in that The device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the data file processing method according to any one of claims 1 to 6.

9. A storage medium, It is characterized in that The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the data file processing method according to any one of claims 1 to 6 are implemented.

10. A computer program product, It is characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the steps of the data file processing method according to any one of claims 1 to 6 are implemented.

Images