Data processing method, device, storage medium and electronic device

Abstract

The embodiment of the present application discloses a data processing method, device, storage medium and electronic device, wherein the method comprises: obtaining a target data stream from a data source, writing the target data stream into a data lake and synchronously performing file merging processing on small data files in the data lake. The embodiment of the present application can improve data processing efficiency.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a data processing method, device, storage medium and electronic device. Background Art

[0002] A data lake refers to a system that uses large binary objects or file formats to store data. It is used to uniformly store data, including both original copies in the original system and converted data, such as reports, visualization data, etc.

[0003] In the big data scenario, due to different business scenarios and data scales, there are various big data engines used to process the needs of a certain field. Common ones can be based on message middleware to store data such as logs, orders, tickets, etc., and then use data engines to consume data and combine with data lakes to achieve data management. Summary of the invention

[0004] The present application provides a data processing method, device, storage medium and electronic device, and the technical solution is as follows:

[0005] In a first aspect, an embodiment of the present application provides a data processing method, the method comprising:

[0006] Get the target data stream from the data source;

[0007] The target data stream is written into the data lake, and small data files in the data lake are simultaneously merged, where the small data files are data files whose file memory is less than a memory threshold.

[0008] In a second aspect, an embodiment of the present application provides a data processing device, the device comprising:

[0009] A data acquisition module is used to acquire a target data stream from a data source;

[0010] A write-merge module is used to write the target data stream into the data lake and simultaneously perform file merging processing on small data files in the data lake, where the small data files are data files whose file memory is less than a memory threshold.

[0011] In a third aspect, an embodiment of the present application provides a computer storage medium, wherein the computer storage medium stores a plurality of instructions, wherein the instructions are suitable for being loaded by a processor and executing the above-mentioned method steps.

[0012] In a fourth aspect, an embodiment of the present application provides an electronic device, which may include: a processor and a memory; wherein the memory stores a computer program, and the computer program is suitable for being loaded by the processor and executing the above-mentioned method steps.

[0013] The beneficial effects brought about by the technical solutions provided by some embodiments of the present application include at least:

[0014] In one or more embodiments of the present application, by obtaining the target data stream in the data source, writing the target data stream into the data lake, and synchronously performing file merging processing on the small data files in the data lake, by adopting a parallel execution method to synchronously write the data stream into the data lake and merge the files, the accumulation of data merging can be avoided, the data processing pressure in the checkpoint stage can be reduced, the data stream processing writing process can be optimized, and the data processing efficiency can be greatly improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0016] Figure 1 It is a flowchart of a data processing method provided in an embodiment of the present application;

[0017] Figure 2 This is a schematic diagram of a data writing scenario provided by an embodiment of the present application;

[0018] Figure 3 It is a flowchart of another data processing method provided in an embodiment of the present application;

[0019] Figure 4 This is a schematic diagram of a file merging scenario provided by an embodiment of the present application;

[0020] Figure 5 This is a comparative analysis diagram of data lake performance provided in an embodiment of the present application;

[0021] Figure 6 It is a flowchart of another data processing method provided in an embodiment of the present application;

[0022] Figure 7a is a performance diagram related to Bloom filtering provided by an embodiment of the present application;

[0023] Figure 7b It is a performance diagram involving hash filtering provided by an embodiment of the present application;

[0024] Figure 7c It is a performance diagram of an open source default filtering method provided by an embodiment of the present application;

[0025] Figure 8is a structural schematic diagram of another data processing device provided in an embodiment of the present application;

[0026] Fig. 9 is a structural diagram of a write-merge module provided in an embodiment of the present application;

[0027] Fig.10 is a structural schematic diagram of another data processing device provided in an embodiment of the present application;

[0028] Fig.11 is a structural diagram of a write-merge module provided in an embodiment of the present application;

[0029] Fig.12 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0030] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0031] In the description of the present application, it should be understood that the terms "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying relative importance. In the description of the present application, it should be noted that, unless otherwise clearly specified and limited, "including" and "having" and any of their variations are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally also includes steps or units that are not listed, or optionally also includes other steps or units inherent to these processes, methods, products or devices. For those of ordinary skill in the art, the specific meanings of the above terms in the present application can be understood in specific circumstances. In addition, in the description of the present application, unless otherwise specified, "multiple" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that there can be three relationships, for example, A and / or B, which can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the associated objects before and after are an "or" relationship.

[0032] In related technologies, it is common to store data such as logs, orders, tickets, etc. based on message middleware, and then use data engines to consume data and implement data management in combination with data lakes. It usually supports near real-time writing and updating, and supports various write query engines such as hive engine, spark engine, and flink engine. It is a data storage architecture based on data lakes.

[0033] Currently, the main data lakes are Hudi Data Lake and Iceberg Data Lake;

[0034] 1) Data engine writing solution based on Hudi data lake: In the message processing stage, the data to be written will be cached first, and then the data will be written to the file in the checkpoint stage. At the same time, when processing data writing, the copyOnWrite method is used to directly merge and rewrite the files to generate a complete new column storage file. This process also completes the function of merging small files. Then, based on the solution of Hudi data lake, first, when processing messages, only data is cached, and written during the checkpoint period, which does not maximize the write performance; secondly, the checkpoint stage completes data writing and small file merging at the same time, and a large number of files need to be updated and read and written. The checkpoint stage has a large amount of tasks and the data processing efficiency is not high.

[0035] 2) Data engine writing solution based on iceberg data lake: Using streaming incremental data writing method, the incremental data is directly written to the corresponding column storage file in real time during the message processing stage, and only the written data is submitted to modify the corresponding metadata file during the checkpoint stage. The data lake uses a mergeOnRead method when writing data, that is, the incremental file is written first, and then the file merge operation is performed, which requires multiple data processing stages to complete; using this method, if the file is not merged for a long time, there will be more incremental files, and there will be a long reading time and memory overflow, resulting in a large amount of data processing tasks and low data processing efficiency.

[0036] The present application is described in detail below with reference to specific embodiments.

[0037] In one embodiment, Figure 1As shown, a data processing method is proposed, which can be implemented by a computer program and can be run on a data processing device based on the von Neumann system. The computer program can be integrated in an application or run as an independent tool application. The data processing device can be a service platform. The service platform can be a separate server device, such as a rack-type, blade, tower, or cabinet-type server device, or a workstation, a large computer, or other hardware device with strong computing power; or it can be a server cluster composed of multiple servers, and each server in the service cluster can be composed in a symmetrical manner, wherein each server has equivalent functions and status in the transaction link, and each server can provide services to the outside independently, and the independent service can be understood as not requiring the assistance of other servers.

[0038] Specifically, the data processing method includes:

[0039] S101: Acquire a target data stream from a data source;

[0040] It can be understood that the data source can be a business library, a tracking log, an application platform or other data source. The data in the data source can be extracted using a data extraction mechanism based on the actual application scenario, such as extracting it once a day. In some embodiments, a message middleware can be used to monitor the Binlog of the data source, and the data of the data source can be accessed to the message middleware in real time. Tracking logs are generally saved in the form of files, which can be extracted regularly using Flume, or accessed in real time using Spark Streaming or Storm. Other data sources are diverse and related to specific businesses, so they will not be described in detail.

[0041] In one or more embodiments of the present specification, the service platform generally obtains data from the data source in the form of a data stream, that is, the service platform can obtain the target data stream in the data source based on the data extraction mechanism in the actual application scenario. In some implementation scenarios, massive data can be entered into the data lake in real time based on the target data stream, and the data lake can be used to store massive amounts of data in different formats.

[0042] In a feasible implementation, the service platform can write the streaming transaction (business) data received by the client into the message middleware, for example, receiving the streaming data from the client through an HTTP receiver. Among them, the client can specifically be an application or web page on an electronic device such as a personal computer or a mobile device, and the streaming transaction (business) data can specifically be point-of-sale data and user browsing data. Then, the streaming transaction (business) data can be written into the Kafka message middleware through the HTTP receiver. For example, real-time data files can be stored through the message middleware, including data files such as logs, orders, and tickets. The streaming computing engine (such as Flink and Spark) consumes data through the message middleware to obtain the streaming transaction (business) data in the data source and writes it into the data lake for storage management, that is, the target data stream. The target data usually contains several data files and usually exists as a set of ordered byte data sequences with a starting point and an end point.

[0043] S102: writing the target data stream into a data lake, and simultaneously performing file merging processing on small data files in the data lake, wherein the small data files are data files whose file memory is smaller than a memory threshold.

[0044] The data lake supports a variety of streaming computing engines for writing queries and other functions. The streaming computing engine can be a Flink engine, a Spark engine, etc. The data lake can be a Hudi data lake, an Iceberg data lake, etc. The data lake is mainly based on the storage architecture of HDFS (distributed file system), supports near real-time writing and updating, and mainly uses columnar storage formats: such as parquet, orc format, etc. The data lake has the advantages of large compression ratio, small storage space, fast reading, and is suitable for olap analysis.

[0045] In one or more embodiments of this specification, "writing the target data stream into the data lake" and "merging small data files in the data lake" can be performed in parallel, without having to wait until several data files in the target data stream are written into the data lake before performing file merging. Through the data processing method of this specification, functions such as incremental writing of file data by the streaming computing engine and file merging can be performed in parallel, greatly improving data processing efficiency.

[0046] In a feasible implementation, when processing data files in a target data stream, a parallel processing task for the target data stream can be created, and the stream computing engine can be controlled to respond to the parallel processing task, write several data files in the target data stream into a table in the data lake while simultaneously merging small data files in the data lake. In this way, it is possible to avoid the need to deploy two data processing tasks for incremental data writing and file merging for the data lake. At the same time, if the file merging takes too long, and there is a high probability that multiple write checkpoints will be submitted during the checkpoint stage involving the data file, data errors will also occur. The data processing method involved in this specification can deploy a data processing task for the stream computing engine (such as deploying a Flink task for parallel processing for the Flink stream computing engine) to complete incremental data writing and file merging, while not affecting the reading performance of the data lake.

[0047] In a specific implementation scenario, the service platform obtains the target data stream and writes the data files in the target data stream into the data lake one by one in real time. During the process of writing into the data lake, if there is at least one data file in the completed writing state, that is, at least one data file in the target data stream has been completed and written into the data lake, which is equivalent to having written incremental data to the data lake, the service platform can synchronously merge the small data files in the data lake when at least one data file has been completed and entered into the lake (that is, completed and written into the data lake).

[0048] Among them, a small data file is a data file whose file memory is less than the memory threshold;

[0049] In daily applications, many small files are generated from daily files generated in personal applications to web applications. Especially in actual transaction scenarios, the rise of blogs, wikis, and spaces has led to changes in the way the Internet provides content. The data based on the Internet is massive, diverse, and dynamically changing, which will also generate a large number of small data files, such as user avatars, album thumbnails, log files, introduction information, etc. Usually, small data files can be understood as data files smaller than a certain memory threshold. The memory threshold can be set based on the actual application environment. For example, the memory threshold can be 1M, 10M, etc. The service platform can determine the small data files smaller than the memory threshold by scanning the data files in the data lake.

[0050] It can be understood that after the service platform completes writing at least one data file in the target data stream into the data lake, these "at least one data file" are also in the completed writing state. In one or more embodiments of the present specification, a file quantity threshold can be set. When the written data files reach the file quantity threshold, the scanning of small data files in the data lake is synchronously started to determine at least one small data file.

[0051] Optionally, scanning of small data files in the data lake is enabled, and the file scanning objects may be all or part of the data files in the data lake. In one or more embodiments, scanning of small data files in the data lake may be performed in an incremental scanning manner, that is, only small data files in the target data stream that are in a write-completed state are scanned.

[0052] It can be understood that synchronously performing file merging processing on the small data files in the data lake may be: by scanning the corresponding files in the data lake, at least determining all the data files (that is, small files) that need to be merged, to generate a file merging task;

[0053] In a specific implementation scenario, the data lake stores data files as a batch of files in the data lake encapsulated into at least one table with business or transactional significance, that is, the data lake stores data files corresponding to several data lake tables. Each time the writing of at least one data file in the target data stream to the partition of the data lake is completed, the data lake table will be updated to complete the record of the incremental data writing. After the data lake completes writing at least one data file, it will submit the data lake table. Usually, the corresponding files in the data lake can be scanned by scanning the data lake table, and the files corresponding to the data lake table are synchronously scanned to filter out the small data files that need to be merged in the data lake table, so as to determine the file merging task. The file merging task at least includes the small data files that need to be merged. It can be understood that the small data files in the data lake are synchronously merged based on the file merging task to generate at least one new target merged file. In some implementation scenarios, after the file merging process is completed to generate the target merged file, since multiple small data files have been merged, it also includes filtering the expired files in the data lake (which can be understood as the small data files that have completed the merger) to complete the data file update of the data lake.

[0054] In a specific implementation scenario, the data lake is taken as an example of the Iceberg data lake to further explain the data lake data update. After each data update (such as incremental update) in the data lake, the corresponding data lake table is updated and committed. Commit will generate a snapshot (Snapshot), which contains at least a unique snapshotId, timestamp timestamp and the corresponding manifest file. The latest snapshot has a global view of the table (data lake table). Each snapshot contains multiple manifest metadata files. Each manifest file records the correspondence between the written files and partitions in this transaction, and contains some statistical information of file records, such as lower_bound, upper_bound, added_rows_count, deleted_rows_count, which is used to quickly filter files. In some implementation scenarios, the manifest file can be understood as an index file.

[0055] It is understandable that the data lake is based on the snapshot feature. The data in iceberg can be accessed through the snapshot. If the data is written to the data lake but not committed successfully, no new snapshot will be generated, and therefore the incomplete data will not be accessed. The files are laid out on the data lake and include at least two parts: the database part and the table part. Each table contains two directories. The first directory metadata stores snapshot and manifest files; the second directory data stores data files, which are divided according to partitions.

[0056] Schematically, writing at least one data file included in the target data stream into the data lake can be implemented by at least one operator for writing data into the data lake. Figure 2 As shown, Figure 2 This is a schematic diagram of a data writing scenario involved in this application. Figure 2 In the example, the operators used to write data into the data lake may include at least: StreamWriter: stream writing operator, FilesCommitter: file submission operator. The number of the above operators can be multiple;

[0057] Schematically, a keyby operation is performed from the data source according to the current table (that is, the data lake table) to enter the StreamWriter operator; the keyby operation (used for stream computing) does not change the data structure of each element of the data, but only re-divides the subtasks for processing data writing according to the key for several data files involved in the input data stream (that is, the target data stream). Through several subtasks, parallel data writing of several data files can be achieved.

[0058] The StreamWriter operator is instantiated through table, and then a taskWriterFactory is constructed. When starting to write data, a writer (which can be understood as an object) is instantiated to perform the writing operation, that is, to write the data in the assigned data file, which is mainly used to write records to the corresponding avro, parquet, orc files. The writer starts writing the data in the source data to the data lake, and records the write operation for the data file at the same time, and then sends the write result WriteResult to the FilesCommitter operator; the write result WriteResult can be understood as the output of the StreamWriter operator by transforming the data file of the target data stream to generate the data (lake) file generated by the process, mainly including the data file when the data file performs the insert operation and the DeleteFile when the delete operation is performed. It can be understood that at this time, the actual written data file is already in the data lake, and the write result WriteResult here only records the metadata of the file, such as location, file name, etc.

[0059] The FilesCommitter operator receives the corresponding data WriteResult, and then collects all DataFile data files when the checkpoint arrives, and submits the Transaction to the data lake to complete the data writing of this checkpoint. In some implementation scenarios, submitting the Transaction will submit the updated data lake table. Schematically, FilesCommitter plays back data files from multiple WriteResults sent by StreamWriter, creates or updates the corresponding type of ManifestFile (file list table) for submission according to different data types (DAtaFile, DeleteFile), and saves the ManifestList as a snapshot in the metadata.json file (which can be understood as a metadata file) with an increasing sequence number.

[0060] Schematically, the FilesCommitter operator maintains a list of DataFile files for each checkpointId (checkpoint), and each checkpointId (checkpoint) corresponds to a checkpointIdid (checkpoint identifier). In this way, even if a transaction of a checkpoint fails to be committed or processed, its original DataFile file is still maintained in the State (the state of the data lake) and can still be committed through subsequent checkpoint rollback.

[0061] In a feasible implementation, the data lake may be based on the Iceberg data lake, and the data processing engine may be the Flink streaming computing engine.

[0062] It can be understood that the above process is continuously carried out when writing the target data stream into the data lake, that is, each data file in the target data stream is continuously written into the data lake through the above process. In the process of writing into the data lake, if there is at least one data file in the completed writing state, that is, at least one data file in the target data stream is completed and written into the data lake, for the data lake, it is equivalent to having written incremental data but not having completed writing all the target data streams. The service platform can perform file merging processing on the small data files in the data lake based on at least one operator synchronization when at least one data file has been completed and entered into the lake (that is, completed writing into the data lake);

[0063] In some embodiments, the updated data lake table is submitted and a (feedback) message can be generated. The message can at least include table name information (such as table name information in the data lake table) and checkpointIdid (checkpoint identifier). The message can be used to at least provide feedback that during the synchronous scanning of small data files in the data lake file, the files recorded in the corresponding table can be scanned based on the table name information.

[0064] In an embodiment of the present application, by obtaining the target data stream in the data source, writing the target data stream into the data lake, and synchronously performing file merging processing on the small data files in the data lake, and by adopting a parallel execution method to synchronously write the data stream into the data lake and merge the files, the accumulation of data merging can be avoided, the data processing pressure in the checkpoint stage is reduced, the data stream processing writing process is optimized, and the data processing efficiency is greatly improved.

[0065] See also Figure 3 , Figure 3 This is a flow chart of another embodiment of a data processing method proposed in this application. Specifically:

[0066] S201: Acquire the target data stream in the data source;

[0067] For details, please refer to the steps of other embodiments in this application specification, which will not be repeated here.

[0068] S202: writing all data files in the target data stream into the data lake respectively;

[0069] For details, please refer to the steps of other embodiments in this application specification, which will not be repeated here.

[0070] S203: If at least one of the data files is in a write-completed state, determining at least one data processing operator, and determining a file merging task through the data processing operator;

[0071] The at least one data processing operator is used to synchronously merge small data files in the data lake. The number of data processing operators can be multiple; the types of each data processing operator can be different so that the user can complete different computing processing instructions.

[0072] For example, operators can refer to task allocation work or processing categories in advance and may include various types of operators, such as scan operators, filter operators, hash operators (for hash join and aggregation), sort operators, input operators, output operators, join operators, and agg operators. Of course, the above are just a few examples of operator types, and there may be other types of operators, which are not limited to this.

[0073] In a feasible implementation, the file merging task determined by the data processing operator may include at least a first processing operator, a second processing operator, and a third processing operator. The first processing operator may be used for file scanning and task generation, for example, the first processing operator may be FileScanTasks, and the first processing operator is a file scanning / task generation type operator; the second processing operator is used for file merging, reading data and writing to a newly merged file, for example, the second processing operator may be RowDataRewriter, used for row data rewriting or data file merging; the third processing operator is used for file replacement and submission of small file merging, and may also be used for expired file deletion, and the third processing operator may be ReplaceDataFiles, and the ReplaceDataFiles operator is used for data file replacement.

[0074] Illustratively, the data lake may be subjected to file scanning processing by a first processing operator to generate a file scanning result, and at least one file merging task may be generated based on the file scanning result.

[0075] Optionally, a start condition may be configured for the first processing operator. The start condition may be the number of completed writes of the data file to the data lake, or the number of (feedback) messages received after the data file is written to the data lake. The start condition is set to start a scan for merging small files when a preset condition is met. In one or more embodiments of the present specification, when the start condition is met, the first processing operator starts scanning the files recorded in the corresponding table to at least filter out small data files, and generates a file merging task based on the scanning result.

[0076] In some embodiments, the delete file information can also be scanned to delete expired files after small file merging to update the file data in the data lake. It can be understood that the file merging task can include all small data files that need to be merged, table name information, and the number of merges in this task. In some embodiments, the file merging task can also include delete file information and / or checkpointId.

[0077] Optionally, the first processing operator may be further configured to improve stability during data processing;

[0078] In a feasible implementation manner, the data processing status of the data table corresponding to the file merging task may be detected;

[0079] The data processing state includes an idle state and a working state. The idle state means that there are no unfinished file merging tasks for the files in the table; the working state means that there are unfinished file merging tasks for the files in the table.

[0080] Optionally, if the data processing state is an idle state, the data processing operator is used to synchronously perform file merging processing on the small data files in the data lake based on the file merging task, and the merging trigger time of the data table corresponding to the file merging task is recorded;

[0081] Optionally, if the data processing status is a working status, the current file merging task is canceled. Canceling the file merging task may be deleting the task, or monitoring whether the last file merging task is completed, and if completed, executing the start file merging task.

[0082] Illustratively, the first processing operator can be configured to detect whether there is an unfinished file merge task for files in at least one of the current tables, that is, whether it is in a working state. If there is an unfinished file merge task, the file merge task is canceled. If there is no unfinished file merge task, that is, it is in an idle state, the file merge task is continued. For example, if a file merge is started, the start time will be recorded. Each time a merge is started, the merge submission time of the latest file merge submission (corresponding to the current table) will be obtained first, that is, it is detected whether there is an unfinished file merge task for files in at least one of the current tables. If the merge submission time is less than or equal to the start time, it means that the last file merge of the current table has not been completed, and the file merge will be canceled to ensure that only one file merge task is executed for each table at the same time, thereby improving stability.

[0083] In a specific implementation scenario, a write message corresponding to each of the data files is obtained through a first processing operator; a write message is a message generated after the data file is written into the data lake, such as a write message is a feedback message in one or more embodiments of this specification. The write message may at least include table name information (such as table name information in the data lake table) and checkpointIdid (checkpoint identifier), and the message may be used to at least feedback that the files recorded in the table can be scanned based on the table name information during the synchronous scanning of small data files in the data lake file.

[0084] The first processing operator performs file scanning processing on the data lake based on the table name information in the write message, generates a file scanning result, and generates at least one file merging task based on the file scanning result; the file scanning result at least includes a scanning result for small data files, such as the number of small data files, the name of small data files, etc. In some embodiments, the file scanning result may also include delete file information (file information corresponding to the delete operation recorded during the write process) and / or checkpointId.

[0085] Schematically, scanning the corresponding files in the data lake can usually be performed by scanning the data lake table, determining the data lake table to be scanned and the data lake files recorded in the data lake table based on the table name information in the write message, and then synchronously scanning the corresponding data lake files in the aforementioned indicated data table based on the table name information to filter out the small data files that need to be merged corresponding to the data lake table, thereby determining the file merging task. The file merging task includes at least the small data files to be merged. It can be understood that the small data files in the data lake are synchronously merged based on the file merging task to generate at least one new target merged file. In some implementation scenarios, after the file merging process is completed to generate the target merged file, since multiple small data files have been merged, it also includes filtering expired files in the data lake (which can be understood as small data files that have completed the merger) to complete the data file update of the data lake.

[0086] S204: Based on the file merging task, the data processing operator is used to synchronously perform file merging processing on the small data files in the data lake.

[0087] In a specific implementation, synchronously merging small data files in the data lake may be performed by at least one data processing operator for file merging. The aforementioned first processing operator is used to determine at least one file merging task, and subsequent file merging may be performed by other data processing operators;

[0088] Indicative, such as Figure 4 As shown, Figure 4 This is a schematic diagram of a file merging scenario involved in the present application. The file merging task determined by the data processing operator may include at least a first processing operator, a second processing operator, and a third processing operator. The first processing operator may be used for file scanning and task generation. For example, the first processing operator may be FileScanTasks, and the first processing operator is a file scanning / task generation operator. The second processing operator is used for file merging, reading data and writing it to the newly merged file. For example, the second processing operator may be RowDataRewriter, which is used for row data rewriting or data file merging. The third processing operator is used for file replacement and submission of small file merging, and can also be used for expired file deletion. The third processing operator may be ReplaceDataFiles, and the ReplaceDataFiles operator is used for data file replacement.

[0089] Illustratively, the data lake is subjected to file scanning processing by the first processing operator FileScanTasks to generate a file scanning result, and at least one file merging task is generated based on the file scanning result.

[0090] Schematically, at least one file merging task can be distributed to at least one second processing operator RowDataRewriter through the first processing operator; Schematically, the second processing operator can receive one or more file merging tasks, which can be configured based on actual conditions. Multiple second processing operators can process several file merging tasks in parallel, and merge small files indicated by the tasks based on the file merging tasks.

[0091] In actual applications, in the incremental update scenario involving the data lake, a streaming computing engine such as the flink engine continuously outputs data to the underlying file system based on the moving time interval to write it into the data lake. Each incremental update of the streaming computing engine will generate a corresponding small data file. The small data file can be a newly written data file, metadata file, snapshot file, etc. These large numbers of small data files need to be merged. In the relevant application scenario, a small file merging method can be set to indicate how the processing operator merges several small data files. For example, the small file merging method can be file compression; it can be a combination of multiple small files according to a fixed merging result, such as merging the small files (small files in the non-cache area) that have been written to the data lake table into a Mapfile in the form of a key value set of the structure "key+value", where the key is the file name of the small file, and the file content of the small file is used as the value. It is spliced ​​according to "key+value", and multiple small files are merged into the key value set to generate a new file as a target merged file. It can be understood that in this application, the small file merging method can be configured based on the actual application environment, and is not specifically limited here.

[0092] Illustratively, when there are multiple second processing operators, each of the second processing operators may execute the file merging task in parallel to synchronously perform file merging processing on the small data files in the data lake to obtain at least one target merged file;

[0093] Schematically, the second processing operator: RowDataRewriter is responsible for performing file merging, reading data and writing it into the newly merged file. RowDataRewriter can execute multiple tasks in parallel. The file merging task will be distributed to each RowDataRewriter for execution. Each RowDataRewriter merges all the small data files that need to be merged based on the file merging task to generate a target merged file.

[0094] Optionally, a thread pool can be opened in the second processing operator RowDataRewriter to execute the file merging task. Each time a file merging task is received, it will be placed in the thread task queue of the thread pool for sequential execution, that is, the file merging task is written into the thread task queue, and the small data files in the data lake are merged based on the thread task queue. After the file merging task is completed, a result (including the newly generated target merged file, the file before the merge, and other information) will be returned and then sent to the downstream ReplaceDataFiles operator.

[0095] In an illustrative manner, the second processing operator can implement multi-threaded parallel execution of "checkpoint processing" and "file merging processing of small data files in the data lake based on the thread task queue" by writing the file merging task into the thread task queue, so as to avoid checkpoint failure caused by the operator being unable to process the checkpoint message due to long task execution time in some scenarios;

[0096] Among them, checkpoint processing can be understood as a checkpoint operation (checkpoint operation), also known as a checkpoint mechanism. Through this checkpoint mechanism, the data lake and the write module (for example, the stream computing processing engine, the Flink cluster) can be guaranteed to restore the state of the entire transaction flow graph to a state before the failure when a certain operator fails for some reason, thereby ensuring the consistency of the transaction flow graph state. It can be understood that the checkpoint mechanism is an important disaster recovery mechanism to ensure data or transaction consistency in scenarios involving incremental writes in the data lake.

[0097] Specifically, the checkpoint operation can save the state of the running task, which includes the state of each operator in the task, the offset of the message in the data source, etc. It should be noted that the checkpoint operation is the core function of implementing the fault tolerance mechanism in the incremental data scenario involved in the data lake. It can generate SnapShots based on the state of each Operator / task in the Stream periodically according to the configuration, so as to regularly persist and store these state data. When the program crashes unexpectedly, it can be selectively restored from these snapshots when rerunning the program, so as to correct the program data anomalies caused by the failure. One of the core concepts of snapshots is barriers. These barriers are injected into the data stream and flow down as part of the data stream together with the data (or records) in the data stream. Barriers will never exceed records, and the data stream is strictly ordered. Barriers isolate the records in the data stream into a series of record sets, and add some data in the set to the current snapshot, while other data is added to the next snapshot. Each barrier carries the ID of the snapshot, and all records before the barrier enter the snapshot. Based on this, in this specification, in order to avoid the checkpoint failure caused by the operator being unable to process the checkpoint message due to the long task execution time during the process of writing to the data lake and merging small files synchronously, the thread task queue is used to assist the second processing operator to perform small file merging and checkpoint processing in parallel. That is to say, when receiving a checkpoint message such as one sent by a streaming computing engine or other operator, the checkpoint message can be processed in parallel by multiple threads based on the thread task queue to perform checkpoint processing on it.

[0098] Furthermore, a third processing operator is used to perform file filtering processing on the data lake based on the at least one target merged file.

[0099] Indicatively, the third processing operator may be a ReplaceDataFiles operator, which is at least used for file replacement, file submission, expired file deletion, etc. After determining to generate the target merged file, the ReplaceDataFiles operator will perform replacement and deletion operations on the small data files that need to be merged. The ReplaceDataFiles operator can record the received completed file merge tasks according to the table name information, checkpointId, and the number of tasks that need to be completed;

[0100] Optionally, you can set the submission condition after the file merge is completed. When the number of tasks of the file merge task is equal to the task number threshold, submit the file merge and update the file storage path in the data lake table. After submission, expired file deletion will be performed to delete the merged small data files and expired snapshot files to complete the file filtering process.

[0101] For reference Figure 5 , Figure 5 This is a comparative analysis chart of data lake performance. Figure 5 In this paper, the open source data lake hudi data lake is used as the comparison object. After optimizing the incremental writing process of the data lake using the data processing method of one or more embodiments of this specification, you can refer to Figure 5 The performance of the "present solution" shown is shown by the dotted line. Figure 5 The vertical axis is the writing performance, in record / s, and the horizontal axis is the amount of data written to the data lake. The optimized solution based on this specification has a stable storage performance of more than 12,000 records / s, while the open source hudi data lake is less than 5,000 records / s. Using the method shown in this specification, after the amount of data written exceeds 10 million, the insert (insert data) and upsert (update data) performance can remain stable, and the data operation stability is good. Both the writing performance and the file merging performance can be taken into account, which is basically not affected by data growth. The writing throughput and parallelism are positively correlated. The writing performance of the data lake is greatly improved, taking into account data writing and performing small file merging at the same time, which greatly improves the data processing efficiency. The write merging parallelism of data processing in this solution is greatly improved.

[0102] In an embodiment of the present application, by obtaining the target data stream in the data source, writing the target data stream into the data lake, and synchronously performing file merging processing on the small data files in the data lake, by adopting a parallel execution mode to write the data stream into the data lake and perform file merging synchronously, it is possible to avoid the accumulation of data merging volume, reduce the data processing pressure in the checkpoint stage, and greatly improve the data processing efficiency; and, by adding multiple processing operators, file merging is performed synchronously when the data stream is written, without having to merge small files in other non-data writing stages after the data stream is completely written, thereby optimizing the data stream processing writing process; in the data lake application layer scenario, computing engines such as Flink only need to deploy one task computing processing task to simultaneously complete the functions of data writing and file merging, without affecting the reading performance.

[0103] See also Figure 6 , Figure 6 This is a flow chart of another embodiment of a data processing method proposed in this application. Specifically:

[0104] S301: Acquire the target data stream in the data source;

[0105] S302: Writing the target data stream into a data lake;

[0106] S303: If at least one of the data files is in a write-completed state, a file merging process is performed on the small data files in the data lake to obtain at least one target merged file.

[0107] For details, please refer to the steps of other embodiments in this application specification, which will not be repeated here.

[0108] S304: Obtain the merge trigger time corresponding to the target merge file;

[0109] The merge trigger time may be understood as the start time for executing the file merge, or the trigger time for starting the file merge.

[0110] In one or more embodiments of the present specification, a file merging task can generally be determined, and based on the file merging task, small data files in the data lake are synchronously processed for file merging, and the task generation time of the file merging task can be used as the merging trigger time.

[0111] S305: Setting a file sequence number of the target merged file based on the merge trigger time, and performing file filtering processing on the data lake based on the file sequence number.

[0112] It can be understood that each time data is incrementally written to the data lake or a file is submitted after a small file is merged, a file sequence number will be generated to mark the new file (such as the target merged file). The file sequence number can be expressed as seqNum, and the file information of the file usually contains the file sequence number.

[0113] In one or more embodiments of the present specification, the data processing stage is to write the data files in the target data stream into the data lake and merge the small data files simultaneously in parallel. For the data lake, after completing the small data file merger to generate the target merged file, the data file will be read to filter out data deletion data that is greater than the current file sequence number. The data deletion data can be the equity-delete data generated when the deletion operation is performed on expired files and merged small data files. The order of file merging and incremental writing submission is not necessarily at the same time point. At this time, if the target merged file is generated and the file sequence number seqNum is generated based on the file generation time corresponding to the target merged file, some data that should be filtered will not be filtered out, resulting in data redundancy and even data recording errors.

[0114] For example, the trigger for merging small files is usually configured. For example, if the checkpoint is started after a specified number of data files are written to the data lake and submitted, then the file submission may be completed in one checkpoint phase, or it may span multiple checkpoint phases to complete the file submission. If the target merged file is generated across several checkpoint phases and the seqNum is generated based on the file generation time, the data of the small data files that have completed the data merge will usually not be recorded in the delete file during the file merge, which will cause the subsequent data that should be filtered to not be filtered out, resulting in data redundancy and even data recording errors.

[0115] In some embodiments, the file submission process corresponding to generating a new file (such as incrementally writing a new file, merging small data files to generate a target merged file) is completed after the checkpoint processing, and the data consistency is verified through the checkpoint processing operation.

[0116] In one or more embodiments of the present specification, by obtaining the merge trigger time corresponding to the target merge file each time the target merge file is generated, the merge trigger time rather than the file generation time is set as the file sequence number of the target merge file, so that when filtering is performed later, at least an equilibrium-delete filter will be performed on it, and the equilibrium-delete filter will delete all data greater than the current file sequence number (equilibrium-delete). Reading the data file can correctly perform file filtering on the data lake based on the file sequence number. All the equilibrium-delete record files processed by the subsequent checkpoint can be filtered out, which can ensure the stability of data processing.

[0117] S306: Obtain a reference checkpoint identifier corresponding to the target merged file, and record the reference checkpoint identifier.

[0118] It is understandable that when a streaming computing engine such as flink writes incremental data files to submit, it will first write an avro file (a binary serialized file format) that records the data file this time. The avro file can be a manifest file, and then record the avro file to the state backend of the streaming computing engine, and record the checkpointId (checkpoint identifier) ​​corresponding to the incremental data file this time, and then complete the checkpoint submission of the current operator. After all operators are completed, the logical checkpoint operation in commitUpToCheckpoint is executed, and the previously written avro file is read out, and the manifest.avro manifest file, snapshot.avro snapshot file and metadata.json metadata file are rewritten, and then the submission is completed. At the same time, the previous avro file will be deleted after the submission is completed. At the same time, the submitted checkpointId will be recorded in the metadata.json metadata file written.

[0119] Checkpoint is used to ensure data consistency. Failure in the checkpoint phase will involve the data recovery process. When the streaming computing engine executes the data recovery process, it will read the checkpointId submitted at the most recent time point from the metadata file, and then only submit checkpoints greater than the checkpointId. However, when files are merged to generate new files and some files are deleted after the subsequent merge, the checkpointId is not written in this process. If the last checkpoint submission belongs to the checkpoint in the file merge scenario when the data recovery process is executed, the submitted checkpoint will be submitted again because the submitted checkpointId is not read. The checkpointId will correspond to the deleted file. At this time, the recovery will fail because the deleted file cannot be found. Here, the submitted checkpointId is written at the same time after the target merge file is generated during the file merge to ensure normal recovery. That is, after the target merge file is generated, the reference checkpoint identifier corresponding to the target merge file is obtained, that is, the checkpoint identifier at this time, and the reference checkpoint identifier is recorded, such as writing the reference checkpoint identifier to the metadata file.

[0120] S307: Obtain a first data volume of the data insertion file and a second data volume of the data deletion file, and perform file filtering processing on the data lake based on the first data volume and the second data volume.

[0121] During the process of incrementally writing data to the data lake, three types of files are written in real time: data files, equilibrium-delete files, and pos-delete files. When performing an insert operation, the data file (that is, the data file newly written to the data lake) is written; when performing a delete operation, the equilibrium-delete file and the pos-delete file are written. The equilibrium-delete file records the entire record value of the record to be deleted, while the pos-delete file records the file and line number of the record to be deleted. When performing an insert operation in each checkpoint stage, the key written in the current checkpoint cycle and the corresponding file line number are recorded in a map.

[0122] The data insertion file is the data file written into the data lake during the insert operation, which can usually be understood as an incremental file. The data insertion file can be the target merged file generated after the merge process.

[0123] The data deletion file is the equilibrium-delete file written when the delete operation is performed. The entire record value of the record to be deleted is recorded in the equilibrium-delete file.

[0124] The first data volume may be understood as the number of files or the total data capacity of the data insertion files.

[0125] The first data volume may be understood as the number of files or the total data capacity of the data deletion files.

[0126] In a specific implementation scenario, in the file merging or reading scenario involving the data lake, file filtering processing of the data lake will be involved, which can usually be based on hash filtering, that is, when reading the equity-delete files corresponding to the file merging stage (which can be understood as expired files after the file merging), all are put into hashest for filtering; in some implementation scenarios, putting all into hashest for filtering will also cause memory overflow.

[0127] Schematically, the hash filtering method corresponds to the file filtering based on HashSet. HashSet has the characteristic of not allowing duplicate elements. By adding all elements of the equilibrium-delete file to the HashSet object, the duplicate elements are automatically removed by utilizing the no-duplicate-element characteristic of the HashSET object.

[0128] In a feasible implementation, in certain scenarios, only the data insertion file can be read, that is, the data file written into the data lake during the insert operation can be filtered to reduce memory usage.

[0129] Indicatively, a threshold can be set. For example, when the amount of data in the equilibrium-delete is n times (such as n is 2) or more than that in the data file, the key filtering of the data file can be used instead of directly reading the equilibrium-delete file. Because in the worst case, all the keys in the data file are in the delete file. At this time, the memory usage of filtering with the data file is equal to the memory usage of the above method. If the amount of data in the equilibrium-delete continues to increase, even in the worst case, the memory usage is less after filtering with the data file. For the case where the amount of data in the file IOequilibrium-delete is twice or more than that of the data file, the impact of reading the data file IO will gradually decrease, which solves the problem of memory overflow.

[0130] Optionally, when the set threshold is reached, the filtering method can be switched, and Bloom filter can be used instead of hash filtering for memory optimization. The key in the data file (usually the key can be the file name) is used to build a bloom filter. If a key is in the data file, the bloom filter will definitely return a correct result. Therefore, bloom filter can be used instead of the hash filtering method corresponding to hashset.

[0131] For example, if the key of the data file occupies 100m of space, the key of the delete file occupies 200m of space, and the key in the data file and the delete file are completely overlapping, then the original hash filtering method is used to directly read the larger memory of the delete file, which usually occupies 200m of memory. Using bloomfilter to read the key of the data file can greatly reduce the memory usage. Generally, its misjudgment rate is very small. For example, if the misjudgment category is only 0.001, the delete file may consume 100.1m of memory when reading. The memory occupied by bloomfilter is generally 1 / 50-1 / 100 of the hash filtering method corresponding to hashset, so the filtering method can be switched.

[0132] In a specific implementation scenario, if the ratio of the first data volume to the second data volume is greater than a target value, a Bloom filtering method is used to perform file filtering on the data lake based on the data insertion file; if the ratio of the first data volume to the second data volume is less than or equal to the target value, a hash filtering method is used to perform file filtering on the data lake based on the data deletion file.

[0133] Optionally, the target data is set based on an actual application scenario. For example, the target value may be set to 2, that is, when the ratio of the first data volume to the second data volume exceeds the target value 2, a Bloom filtering method is used.

[0134] See also Figure 7a-7c , Figure 7a is a performance diagram involving Bloom filtering, Figure 7b is a performance diagram involving hash filtering. Figure 7c A performance diagram of a data lake using open source default filtering; combined Figure 7a and Figure 7b The memory usage of different data filtering methods based on the first data volume and the second data volume using the aforementioned method can be intuitively fed back. Under the condition that other conditions are the same, it can be seen that the memory usage peak of the bloomfilter method is stable at around 2G, the memory usage peak of the hashset method is basically between 2-3G, and the memory usage peak of the original version is basically between 3-4G. That is, the data processing method corresponding to this manual can save memory usage in the data processing stage and save data processing resource consumption.

[0135] In an embodiment of the present application, by obtaining the target data stream in the data source, writing the target data stream into the data lake, and synchronously performing file merging processing on the small data files in the data lake, and by adopting a parallel execution method to synchronously write the data stream into the data lake and merge the files, the accumulation of data merging volume can be avoided, and the data processing pressure in the checkpoint stage can be reduced, the data stream processing writing process can be optimized, and the data processing efficiency can be greatly improved; and while taking into account the data processing efficiency, the normal file filtering process can be guaranteed, file filtering errors can be avoided, and the disaster recovery performance of data processing can be improved.

[0136] The following will be combined Figure 8 , the data processing device provided in the embodiment of the present application is introduced in detail. It should be noted that, Figure 8 The data processing device shown is used to execute the method steps of the embodiment shown in the present application. For the convenience of explanation, only the parts related to the embodiment of the present application are shown. For specific technical details not disclosed, please refer to other embodiments of the present application.

[0137] See also Figure 8 , which shows a schematic diagram of the structure of the data processing device of an embodiment of the present application. The data processing device 1 can be implemented as all or part of the user terminal through software, hardware or a combination of both. According to some embodiments, the data processing device 1 includes a data acquisition module 11 and a write-merge module 12, which are specifically used to:

[0138] The data acquisition module 11 is used to acquire the target data stream from the data source;

[0139] The write-merge module 12 is used to write the target data stream into the data lake and simultaneously perform file merging processing on small data files in the data lake, where the small data files are data files whose file memory is less than a memory threshold.

[0140] Optional, such as Fig. 9 As shown, the write-merge module 12 includes:

[0141] A data writing unit 121 is used to write all data files in the target data stream into the data lake respectively;

[0142] The file merging unit 122 is used to synchronously merge the small data files in the data lake if at least one of the data files is in a write-completed state.

[0143] Optionally, the file merging unit 122 is specifically configured to:

[0144] A file merging task is determined, and based on the file merging task, a file merging process is synchronously performed on the small data files in the data lake.

[0145] Optionally, the file merging unit 122 is specifically configured to:

[0146] At least one data processing operator is determined, a file merging task is determined by the data processing operator, and based on the file merging task, the data processing operator is used to synchronously perform file merging processing on small data files in the data lake.

[0147] Optionally, the file merging unit 122 is specifically configured to:

[0148] The data lake is subjected to file scanning processing by a first processing operator to generate a file scanning result, and at least one file merging task is generated based on the file scanning result.

[0149] Optionally, the file merging unit 122 is specifically configured to:

[0150] Obtaining a write message corresponding to each of the data files through a first processing operator;

[0151] The first processing operator performs file scanning processing on the data lake based on the table name information in the written message to generate a file scanning result.

[0152] Optionally, the file merging unit 122 is specifically configured to:

[0153] Distributing the at least one file merging task to at least one second processing operator through the first processing operator;

[0154] The second processing operators respectively execute the file merging tasks in parallel to synchronously perform file merging processing on the small data files in the data lake to obtain at least one target merged file;

[0155] The data lake is subjected to file filtering processing based on the at least one target merged file by a third processing operator.

[0156] Optionally, the file merging unit 122 is specifically configured to:

[0157] Detecting the data processing status of the data table corresponding to the file merging task;

[0158] If the data processing state is an idle state, the data processing operator is used to synchronously perform file merging processing on the small data files in the data lake based on the file merging task, and the merging trigger time of the data table corresponding to the file merging task is recorded;

[0159] If the data processing status is a working status, the file merging task is canceled.

[0160] Optionally, the write-merge module 12 is specifically configured to:

[0161] Writing the file merging task into the thread task queue;

[0162] The small data files in the data lake are merged based on the thread task queue.

[0163] Optionally, the write-merge module 12 is specifically configured to:

[0164] Receive checkpoint messages;

[0165] The checkpoint message is checked by a parallel execution method, and the small data files in the data lake are merged based on a thread task queue.

[0166] Optionally, the write-merge module 12 is specifically configured to:

[0167] Performing file merging processing on the small data files in the data lake to obtain at least one target merged file;

[0168] Get the merge trigger time corresponding to the target merge file;

[0169] A file sequence number of the target merged file is set based on the merge trigger time, and file filtering processing is performed on the data lake based on the file sequence number.

[0170] Optional, such as Fig.10 As shown, the device 1 further includes:

[0171] A file filtering module 13, used to obtain a first data volume of a data insertion file and a second data volume of a data deletion file;

[0172] The file filtering module 13 is further configured to perform file filtering processing on the data lake based on the first data volume and the second data volume.

[0173] Optional, such as Fig.11 As shown, the file filtering module 13 includes:

[0174] A Bloom filtering unit 131, configured to perform file filtering processing on the data lake by using a Bloom filtering method based on the data insertion file if the ratio of the first data volume to the second data volume is greater than a target value;

[0175] The hash filtering unit 132 is configured to perform file filtering processing on the data lake by using a hash filtering method based on the data deletion file if the ratio of the first data volume to the second data volume is less than or equal to a target value.

[0176] It should be noted that, when the data processing device provided in the above embodiment executes the data processing method, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the data processing device provided in the above embodiment and the data processing method embodiment belong to the same concept, and the implementation process thereof is detailed in the method embodiment, which will not be repeated here.

[0177] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0178] In an embodiment of the present application, in an embodiment of the present application, by obtaining a target data stream in a data source, writing the target data stream into a data lake, and synchronously performing file merging processing on small data files in the data lake, writing the data stream into the data lake and merging files simultaneously are performed in a parallel execution manner, thereby avoiding accumulation of data merging volume, reducing data processing pressure at the checkpoint stage, optimizing the data stream processing writing process, and greatly improving data processing efficiency; and while taking into account data processing efficiency, the file filtering process can be guaranteed to be normal, file filtering errors can be avoided, and the disaster recovery performance of data processing can be improved.

[0179] The present application also provides a computer storage medium, which can store multiple instructions, and the instructions are suitable for being loaded and executed by a processor as described above. Figure 1 to Figure 7c The data processing method of the embodiment shown in the figure can be specifically executed by referring to Figure 1 to Figure 7c The specific description of the illustrated embodiment will not be repeated here.

[0180] The present application also provides a computer program product, which stores at least one instruction, and the at least one instruction is loaded and executed by the processor as described above. Figure 1 to Figure 7c The data processing method of the embodiment shown in the figure can be specifically executed by referring to Figure 1 to Figure 7c The specific description of the illustrated embodiment will not be repeated here.

[0181] See also Fig.12 , provides a schematic diagram of the structure of an electronic device for one or more embodiments of this specification. Fig.12 As shown, the electronic device 1000 may include: at least one processor 1001 , at least one network interface 1004 , a user interface 1003 , a memory 1005 , and at least one communication bus 1002 .

[0182] The communication bus 1002 is used to realize the connection and communication between these components.

[0183] The user interface 1003 may include a display screen (Display) and a camera (Camera), and the optional user interface 1003 may also include a standard wired interface and a wireless interface.

[0184] The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a WI-FI interface).

[0185] Among them, the processor 1001 may include one or more processing cores. The processor 1001 uses various interfaces and lines to connect various parts of the entire server 1000, and executes various functions and processes data of the server 1000 by running or executing instructions, programs, code sets or instruction sets stored in the memory 1005, and calling data stored in the memory 1005. Optionally, the processor 1001 can be implemented in at least one hardware form of digital signal processing (Digital Signal Processing, DSP), field programmable gate array (Field-Programmable Gate Array, FPGA), and programmable logic array (Programmable Logic Array, PLA). The processor 1001 can integrate one or a combination of a central processing unit (Central Processing Unit, CPU), a graphics processor (Graphics Processing Unit, GPU) and a modem. Among them, the CPU mainly processes the operating system, user interface and application programs; the GPU is responsible for rendering and drawing the content to be displayed on the display screen; the modem is used to process wireless communication. It can be understood that the above-mentioned modem may not be integrated into the processor 1001, and it can be implemented separately through a chip.

[0186] Among them, the memory 1005 may include a random access memory (Random Access Memory, RAM) and may also include a read-only memory (Read-Only Memory). Optionally, the memory 1005 includes a non-transitory computer-readable storage medium. The memory 1005 can be used to store instructions, programs, codes, code sets or instruction sets. The memory 1005 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the above-mentioned various method embodiments, etc.; the data storage area may store data involved in the above-mentioned various method embodiments, etc. The memory 1005 may optionally be at least one storage device located away from the aforementioned processor 1001. As Fig.12 As shown, the memory 1005 as a computer storage medium may include an operating system, a network communication module, a user interface module and an application program.

[0187] exist Fig.10In the electronic device 1000 shown, the user interface 1003 is mainly used to provide an input interface for the user and obtain data input by the user; and the processor 1001 can be used to call the application stored in the memory 1005 and specifically perform the following operations:

[0188] Get the target data stream from the data source;

[0189] The target data stream is written into the data lake, and small data files in the data lake are simultaneously merged, where the small data files are data files whose file memory is less than a memory threshold.

[0190] In one embodiment, when the processor 1001 writes the target data stream into the data lake and simultaneously performs file merging processing on the small data files in the data lake, the processor 1001 specifically performs the following operations:

[0191] Writing all data files in the target data stream into the data lake respectively;

[0192] If at least one of the data files is in a writing completion state, the small data files in the data lake are synchronously merged.

[0193] In one embodiment, the processor 1001 performs file merging processing on the small data files in the data lake when executing the synchronization, including:

[0194] A file merging task is determined, and based on the file merging task, a file merging process is synchronously performed on the small data files in the data lake.

[0195] In one embodiment, the processor 1001 performs the determined file merging task and synchronously performs file merging processing on the small data files in the data lake based on the file merging task, including:

[0196] At least one data processing operator is determined, a file merging task is determined by the data processing operator, and based on the file merging task, the data processing operator is used to synchronously perform file merging processing on small data files in the data lake.

[0197] In one embodiment, when executing the task of determining the file merging task, the processor 1001 specifically performs the following steps:

[0198] The data lake is subjected to file scanning processing by a first processing operator to generate a file scanning result, and at least one file merging task is generated based on the file scanning result.

[0199] In one embodiment, when the processor 1001 performs the file scanning process on the data lake through the first processing operator to generate the file scanning result, the processor 1001 specifically performs the following steps:

[0200] Obtaining a write message corresponding to each of the data files through a first processing operator;

[0201] The first processing operator performs file scanning processing on the data lake based on the table name information in the written message to generate a file scanning result.

[0202] In one embodiment, when the processor 1001 performs the file merging process based on the file merging task and synchronously performs the file merging process on the small data files in the data lake using the data processing operator, the processor 1001 specifically performs the following steps:

[0203] Distributing the at least one file merging task to at least one second processing operator through the first processing operator;

[0204] The second processing operators respectively execute the file merging tasks in parallel to synchronously perform file merging processing on the small data files in the data lake to obtain at least one target merged file;

[0205] The data lake is subjected to file filtering processing based on the at least one target merged file by a third processing operator.

[0206] In one embodiment, when the processor 1001 performs the file merging process based on the file merging task and synchronously performs the file merging process on the small data files in the data lake using the data processing operator, the processor 1001 specifically performs the following steps:

[0207] Detecting the data processing status of the data table corresponding to the file merging task;

[0208] If the data processing state is an idle state, the data processing operator is used to synchronously perform file merging processing on the small data files in the data lake based on the file merging task, and the merging trigger time of the data table corresponding to the file merging task is recorded;

[0209] If the data processing status is a working status, the file merging task is canceled.

[0210] In one embodiment, when the processor 1001 performs the file merging process on the small data files in the data lake based on the file merging task synchronization, the processor 1001 specifically performs the following steps:

[0211] Writing the file merging task into the thread task queue;

[0212] The small data files in the data lake are merged based on the thread task queue.

[0213] In one embodiment, after executing the step of writing each of the file merging tasks into the thread task queue, the processor 1001 further performs the following steps:

[0214] Receive checkpoint messages;

[0215] The file merging process of the small data files in the data lake based on the thread task queue includes:

[0216] The checkpoint message is checked by a parallel execution method, and the small data files in the data lake are merged based on a thread task queue.

[0217] In one embodiment, when the processor 1001 performs the file merging process on the small data files in the data lake, the processor 1001 specifically performs the following steps:

[0218] Performing file merging processing on the small data files in the data lake to obtain at least one target merged file;

[0219] Get the merge trigger time corresponding to the target merge file;

[0220] A file sequence number of the target merged file is set based on the merge trigger time, and file filtering processing is performed on the data lake based on the file sequence number.

[0221] In one embodiment, when executing the data processing method, the processor 1001 further performs the following steps:

[0222] Acquire a first data volume of a data insertion file and a second data volume of a data deletion file;

[0223] Based on the first data volume and the second data volume, file filtering processing is performed on the data lake.

[0224] In one embodiment, when the processor 1001 performs the file filtering process on the data lake by using the Bloom filtering method based on the first data volume and the second data volume, the processor 1001 specifically performs the following steps:

[0225] If the ratio of the first data volume to the second data volume is greater than a target value, performing file filtering processing on the data lake by using a Bloom filtering method based on the data insertion file;

[0226] If the ratio of the first data volume to the second data volume is less than or equal to the target value, the data lake is filtered by a hash filter based on the data deletion file. A person skilled in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the storage medium can be a disk, an optical disk, a read-only storage memory, or a random access storage memory, etc.

[0227] In an embodiment of the present application, by obtaining the target data stream in the data source, writing the target data stream into the data lake, and synchronously performing file merging processing on the small data files in the data lake, and by adopting a parallel execution method to synchronously write the data stream into the data lake and merge the files, the accumulation of data merging volume can be avoided, and the data processing pressure in the checkpoint stage can be reduced, the data stream processing writing process can be optimized, and the data processing efficiency can be greatly improved; and while taking into account the data processing efficiency, the normal file filtering process can be guaranteed, file filtering errors can be avoided, and the disaster recovery performance of data processing can be improved.

[0228] The above disclosure is only the preferred embodiment of the present application, which certainly cannot be used to limit the scope of rights of the present application. Therefore, equivalent changes made according to the claims of the present application are still within the scope covered by the present application.

Claims

1. A data processing method, characterized in that: The method comprises: Get the target data stream from the data source; Writing the target data stream into the data lake, and simultaneously performing file merging processing on small data files in the data lake, wherein the small data files are data files whose file memory is smaller than a memory threshold; Acquire a first data volume of a data insertion file and a second data volume of a data deletion file; If the ratio of the first data volume to the second data volume is greater than a target value, performing file filtering processing on the data lake by using a Bloom filtering method based on the data insertion file; If the ratio of the first data volume to the second data volume is less than or equal to a target value, a hash filtering method is used to perform file filtering processing on the data lake based on the data deletion file; wherein the target value includes 2.

2. The method according to claim 1, characterized in that The step of writing the target data stream into the data lake and simultaneously performing file merging processing on the small data files in the data lake includes: Writing all data files in the target data stream into the data lake respectively; If at least one of the data files is in a writing completion state, the small data files in the data lake are synchronously merged.

3. The method according to claim 2, characterized in that The synchronization performs file merging processing on the small data files in the data lake, including: A file merging task is determined, and based on the file merging task, a file merging process is synchronously performed on the small data files in the data lake.

4. The method according to claim 3, characterized in that The determining of the file merging task and synchronously performing file merging processing on the small data files in the data lake based on the file merging task include: At least one data processing operator is determined, a file merging task is determined by the data processing operator, and based on the file merging task, the data processing operator is used to synchronously perform file merging processing on small data files in the data lake.

5. The method according to claim 4, characterized in that The step of determining the file merging task includes: The data lake is subjected to file scanning processing by a first processing operator to generate a file scanning result, and at least one file merging task is generated based on the file scanning result.

6. The method according to claim 5, characterized in that The performing file scanning processing on the data lake by the first processing operator to generate a file scanning result includes: Obtaining a write message corresponding to each of the data files through a first processing operator; The first processing operator performs file scanning processing on the data lake based on the table name information in the written message to generate a file scanning result.

7. The method according to claim 5, characterized in that The step of synchronously performing file merging processing on the small data files in the data lake by using the data processing operator based on the file merging task includes: Distributing the at least one file merging task to at least one second processing operator through the first processing operator; The second processing operators respectively execute the file merging tasks in parallel to synchronously perform file merging processing on the small data files in the data lake to obtain at least one target merged file; The data lake is subjected to file filtering processing based on the at least one target merged file by a third processing operator.

8. The method according to claim 4, characterized in that The step of synchronously performing file merging processing on the small data files in the data lake by using the data processing operator based on the file merging task includes: Detecting the data processing status of the data table corresponding to the file merging task; If the data processing state is an idle state, the data processing operator is used to synchronously perform file merging processing on the small data files in the data lake based on the file merging task, and the merging trigger time of the data table corresponding to the file merging task is recorded; If the data processing status is a working status, the file merging task is canceled.

9. The method according to claim 3, characterized in that: The synchronously performing file merging processing on the small data files in the data lake based on the file merging task includes: Writing the file merging task into the thread task queue; The small data files in the data lake are merged based on the thread task queue.

10. The method according to claim 9, characterized in that After writing each of the file merging tasks into the thread task queue, the method further includes: Receive checkpoint messages; The file merging process of the small data files in the data lake based on the thread task queue includes: The checkpoint message is checked by a parallel execution method, and the small data files in the data lake are merged based on a thread task queue.

11. The method according to claim 1, characterized in that: The file merging process of the small data files in the data lake includes: Performing file merging processing on the small data files in the data lake to obtain at least one target merged file; Get the merge trigger time corresponding to the target merge file; A file sequence number of the target merged file is set based on the merge trigger time, and file filtering processing is performed on the data lake based on the file sequence number.

12. A data processing device, characterized in that: include: A data acquisition module is used to acquire a target data stream from a data source; A write-merge module, used to write the target data stream into the data lake, and simultaneously perform file merging processing on small data files in the data lake, wherein the small data files are data files whose file memory is less than a memory threshold; A file filtering module, used for obtaining a first data volume of a data insertion file and a second data volume of a data deletion file; The file filtering module also includes a Bloom filtering unit and a hash filtering unit. The Bloom filtering unit is used to perform file filtering processing on the data lake by using a Bloom filtering method based on the data insertion file if the ratio of the first data volume to the second data volume is greater than a target value; the hash filtering unit is used to perform file filtering processing on the data lake by using a hash filtering method based on the data deletion file if the ratio of the first data volume to the second data volume is less than or equal to a target value; wherein the target value includes 2.

13. A computer storage medium, characterized in that: The computer storage medium stores a plurality of instructions, and the instructions are suitable for being loaded by a processor and executing the method steps as claimed in any one of claims 1 to 11.

14. An electronic device, characterized in that: include: A processor and a memory; wherein the memory stores a computer program, and the computer program is suitable for being loaded by the processor and executing the method steps as claimed in any one of claims 1 to 11.

Images