A data lake data processing method, device, system, equipment and medium

Abstract

The application relates to the technical field of big data, and provides a data lake data processing method, device, system, equipment and medium. The method is applied to a slave node service, and the method comprises the following steps: obtaining to-be-written data; recording the to-be-written data in a pre-created pre-write log file, and recording the to-be-written data in a pre-created memory partition according to the data type of the to-be-written data; generating an initial table format file corresponding to the to-be-written data according to the information recorded in the pre-write log file and the memory partition, and transmitting the initial table format file to a master node service to generate an initial snapshot; converting an equivalent deletion file in the initial snapshot into a position deletion file, and merging the files in the initial snapshot according to a preset table format file capacity to obtain a target table format file; transmitting the target table format file to the master node service to generate a target snapshot, and completing writing of the to-be-written data into a data lake. Through the application scheme, the efficiency of data lake read-write processing can be improved.

Description

Technical Field

[0001] This application relates to the field of big data technology, specifically to a data lake data processing method, apparatus, system, equipment, and medium. Background Technology

[0002] A data lake is a method of storing data in a system or repository in a natural format that helps to configure data in various schemas and structural forms, typically object blocks or files.

[0003] Currently, data lakes do not provide perfect support for integrated streaming and batch processing, and they do not adequately support the low latency requirements of real-time computing. They only support real-time computing latency down to the minute level, while real-time computing latency down to the second level cannot be met in data lakes.

[0004] Existing data lakes generate numerous small files during snapshot commits, reducing read / write efficiency. Furthermore, data lakes generate a large number of equality-deleted files during data read / write operations, which consume significant memory and further degrade read / write efficiency.

[0005] Therefore, how to improve the efficiency of data reading and writing in data lakes is an urgent problem to be solved. Summary of the Invention

[0006] In view of the shortcomings of the prior art described above, the purpose of this application is to provide a data lake data processing method, apparatus, system, device and medium to solve the problem of how to improve the efficiency of data lake data reading and writing in the prior art.

[0007] To achieve the above and other related objectives, this application provides a data lake data processing method, which is applied to a slave node service, and the method includes:

[0008] Get the data to be written;

[0009] The data to be written is recorded in a pre-created write-ahead log file, and the data to be written is recorded in a pre-created memory partition according to the data type of the data to be written, wherein there is a mapping relationship between the write-ahead log file and the memory partition;

[0010] Based on the information recorded in the write-ahead log file and the memory partition, an initial table format file corresponding to the data to be written is generated, and the initial table format file is transmitted to the master node service to generate an initial snapshot;

[0011] The files in the initial snapshot that are deleted by equal value are converted into files that are deleted by location, and the files in the initial snapshot are merged according to the preset tabular file size to obtain the target tabular file;

[0012] The target table file is transferred to the master node service to generate a target snapshot, thus completing the writing of the data to be written to the data lake.

[0013] In one embodiment of this application, before obtaining the data to be written, the method further includes:

[0014] The heartbeat information of the slave node service is sent to the master node service, so that the master node service generates a list of available slave node services based on the heartbeat information.

[0015] The master node service transmits the list of available slave node services to the client, so that the client can select available slave node services according to the list and issue data processing instructions.

[0016] In one embodiment of this application, selecting an available slave node service based on the available slave node service list and issuing a data processing instruction includes:

[0017] Select multiple available slave node services from the list of available slave node services, perform load balancing processing based on preset Internet Protocol information and the multiple available slave node services, and establish communication connections with the multiple available slave node services;

[0018] Data write instructions and / or data read instructions are sent to each available slave node service in the communication connection to realize data processing of the data lake.

[0019] In one embodiment of this application, the heartbeat information includes the slave node service name, slave node service identity identifier, slave node service Internet Protocol address, and slave node service port information;

[0020] The target table format file includes data file path information, data file serial number, location-deleted file path information, location-deleted file serial number, write-ahead log file, and slave node service identity identifier.

[0021] In one embodiment of this application, recording the data to be written in a pre-created memory partition according to the data type of the data to be written includes:

[0022] When the data type is data to be deleted, the data to be written is recorded in the deletes partition, and the records corresponding to the data to be written in the indexes partition and the List partition are deleted.

[0023] When the data type is update data and the indexes partition contains a record corresponding to the data to be written, the record corresponding to the data to be written in the indexes partition is updated to the List partition;

[0024] When the data type is update data and there is no record corresponding to the data to be written in the indexes partition, the data to be written is recorded in the deletes partition and the list partition, and the record position of the data to be written in the list partition is recorded in the indexes partition;

[0025] When the data type is inserted data, the data to be written is recorded in the List partition;

[0026] The deletes partition, indexes partition, and list partition are included in the pre-created memory partition.

[0027] In one embodiment of this application, when the write-ahead log file is full of data, the write-ahead log file is closed and added to a pre-built write-ahead log file queue;

[0028] When the memory partition is full of data, the write-ahead log file is closed and added to a pre-built write-ahead log file queue.

[0029] If the write-ahead log file queue is empty when the slave node service consumes the preset table format file generation thread, then the write-ahead log file with currently existing data is closed, and the write-ahead log file is added to the write-ahead log file queue.

[0030] In one embodiment of this application, an initial table format file corresponding to the data to be written is generated based on the write-ahead log file and the information recorded in the memory partition, including:

[0031] When the data type is inserted data, the data to be written is transferred to the target data lake table format data file;

[0032] When the data type is data to be deleted, the data to be written is transferred to the equivalent deletion file of the target data lake table format;

[0033] When the data type is update data and it is recorded in the deletes partition, the data to be written is transferred to the data file, the position information of the data to be written in the data file is recorded, and the data to be written is transferred to the equivalent deletion file;

[0034] When the data type is update data and it is recorded in the indexes partition, the data to be written is transferred to the data file, the position information of the data to be written in the data file is recorded, and the position information is recorded in the position deletion file of the target data lake table format.

[0035] In one embodiment of this application, converting the equivalent deletion file in the initial snapshot into a location deletion file includes:

[0036] Based on the sequence number of the equally deleted files in the initial snapshot, the equally deleted files are sorted in ascending order;

[0037] Traverse the equivalent deletion files in ascending order and store the records in the equivalent deletion files into the memory cache;

[0038] Obtain the files to be converted from the data file whose serial numbers are less than the serial numbers of the files to be deleted;

[0039] When the file to be converted exists in the memory cache, a new location deletion file is generated based on the file to be converted, and the serial number of the equivalent deletion file, the path of the data file, and the position of the file to be converted in the data file are added to the new location deletion file.

[0040] In one embodiment of this application, merging files in the initial snapshot according to a preset tabular file size includes:

[0041] Obtain the data files to be merged and the files to be deleted at the merge locations from the initial snapshot whose file size is smaller than the preset first table format file size;

[0042] According to the preset second table format file capacity, multiple data files to be merged are merged, and multiple files to be deleted at the merged location are merged, wherein the capacity of the second table format file is greater than the capacity of the first table format file.

[0043] In one embodiment of this application, after transmitting the target tabular file to the master node service to generate a target snapshot, the method further includes:

[0044] When receiving the target snapshot submission result transmitted by the master node service, the name of the write-ahead log file to be deleted is obtained from the target snapshot submission result.

[0045] Based on the name of the write-ahead log file to be deleted, delete the write-ahead log file in the write-ahead log file queue and delete the data in the memory partition corresponding to the name of the write-ahead log file to be deleted.

[0046] In one embodiment of this application, after writing the data to be written to the data lake is completed, the method further includes:

[0047] In response to a data read command, the read cache function of the target data lake table format is enabled to obtain the target data in the target data lake table format;

[0048] Delete the record corresponding to the target data in the deletes partition, and update the record corresponding to the target data in the List partition to complete the reading of the target data in the data lake.

[0049] In one embodiment of this application, a data lake data processing apparatus is also provided, the apparatus comprising:

[0050] The data acquisition module is used to acquire the data to be written.

[0051] The data recording module is used to record the data to be written in a pre-created write-ahead log file, and to record the data to be written in a pre-created memory partition according to the data type of the data to be written, wherein there is a mapping relationship between the write-ahead log file and the memory partition;

[0052] The initial snapshot generation module is used to generate an initial table format file corresponding to the data to be written based on the write-ahead log file and the information recorded in the memory partition, and to transmit the initial table format file to the master node service to generate an initial snapshot;

[0053] The optimization module is used to convert the files with equal deletion value in the initial snapshot into files with location deletion value, and merge the files in the initial snapshot according to the preset tabular file size to obtain the target tabular file;

[0054] The target snapshot generation module is used to transfer the target tabular file to the master node service to generate a target snapshot, thereby completing the writing of the data to be written into the data lake.

[0055] In one embodiment of this application, a data lake data processing system is also provided, the system comprising:

[0056] The slave node service is used to obtain the data to be written; record the data to be written in a pre-created write-ahead log file; record the data to be written in a pre-created memory partition according to the data type of the data to be written, wherein the write-ahead log file and the memory partition have a mapping relationship; generate an initial table format file corresponding to the data to be written based on the information recorded in the write-ahead log file and the memory partition, and transmit the initial table format file to the master node service to generate an initial snapshot; convert the equal-value deletion files in the initial snapshot into position deletion files, and merge the files in the initial snapshot according to the preset table format file capacity to obtain a target table format file; transmit the target table format file to the master node service to generate a target snapshot, thus completing the writing of the data to be written to the data lake;

[0057] The master node service is used to receive the initial tabular file to generate an initial snapshot, and to receive the target tabular file to generate a target snapshot;

[0058] The client is used to transmit the data to be written to the slave node service.

[0059] In one embodiment of this application, an electronic device is also provided, the electronic device comprising:

[0060] One or more processors;

[0061] A storage device for storing one or more programs, which, when executed by one or more processors, cause the electronic device to implement the data lake data processing method as described above.

[0062] In one embodiment of this application, a computer-readable storage medium is also provided, on which a computer program is stored, which, when executed by a computer's processor, causes the computer to perform the data lake data processing method as described above.

[0063] The beneficial effects of this invention are:

[0064] This invention is applied to the slave node service in a master-slave node service mode. The slave node service first obtains the data to be written; then, it records the data to be written in a pre-created write-ahead log file, and records the data to be written in a pre-created memory partition according to the data type of the data to be written, wherein the write-ahead log file and the memory partition have a mapping relationship; then, based on the information recorded in the write-ahead log file and the memory partition, it generates an initial table format file corresponding to the data to be written, and transmits the initial table format file to the master node service to generate an initial snapshot; then, it converts the equal-value deletion files in the initial snapshot into position deletion files, and merges the files in the initial snapshot according to the preset table format file capacity to obtain a target table format file; finally, it transmits the target table format file to the master node service to generate a target snapshot, completing the writing of the data to be written to the data lake. In this invention, the master node service and slave node service cooperate to achieve data processing in a data lake table format. When writing data, the client can ignore the snapshot generation time of the data lake table format and write the data to the slave node service in real time. Through the write-ahead log file and memory partition with a mapping relationship, data writing can be achieved at the second level, improving the efficiency of data lake writing. The slave node service can enable data reading caching to improve the visibility of data reading in the data lake table format, thereby improving the efficiency of data lake reading. The equivalent deletion files in the initial snapshot are converted into positional deletion files. The data lake table format data files and positional deletion files can be traversed together and processed by key to achieve the effect of data deletion. Eliminating the equivalent deletion files can eliminate the global memory index of the data lake table format data, reduce memory consumption, and improve the efficiency of data lake read and write processing. According to the preset table format capacity, small files in the snapshot are merged to eliminate the number of positional deletion files, further improving the efficiency of data lake read and write processing.

[0065] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0066] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application. It is obvious that the drawings described below are merely some embodiments of this application, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort. In the drawings:

[0067] Figure 1 This is a schematic diagram illustrating the implementation environment of a data lake data processing method according to an exemplary embodiment of this application;

[0068] Figure 2This is a schematic flowchart illustrating a data lake data processing method according to an exemplary embodiment of this application;

[0069] Figure 3 This is a schematic diagram of the structure of a data lake data processing system, as illustrated in an exemplary embodiment of this application.

[0070] Figure 4 This is a schematic diagram of the client structure shown in an exemplary embodiment of this application;

[0071] Figure 5 This is a schematic diagram illustrating the structure of the master node service in an exemplary embodiment of this application;

[0072] Figure 6 This is a schematic diagram illustrating the structure of a slave node service, as shown in an exemplary embodiment of this application;

[0073] Figure 7 This is a schematic diagram illustrating the internal interactions of the master node service, as shown in an exemplary embodiment of this application.

[0074] Figure 8 This is a schematic diagram illustrating the internal interaction of a slave node service, as shown in an exemplary embodiment of this application.

[0075] Figure 9 This is a schematic diagram of a write-ahead log fragmentation and memory partitioning structure shown in an exemplary embodiment of this application;

[0076] Figure 10 This is an exemplary embodiment of the present application illustrating the recording of data to be written to a write-ahead log file and a memory partition;

[0077] Figure 11 This is a schematic diagram illustrating the generation of a tabular file, as shown in an exemplary embodiment of this application;

[0078] Figure 12 This is a block diagram illustrating a data lake data processing apparatus according to an exemplary embodiment of this application;

[0079] Figure 13 A schematic diagram of the structure of a computer system suitable for an electronic device according to an embodiment of this application is shown. Detailed Implementation

[0080] The embodiments of the present invention will be described below with reference to the accompanying drawings and preferred embodiments. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be understood that the preferred embodiments are only for illustrating the present invention and not for limiting the scope of protection of the present invention.

[0081] It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Therefore, the drawings only show the components related to the present invention and are not drawn according to the actual number, shape and size of the components in the actual implementation. In the actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0082] In the following description, numerous details are explored to provide a more thorough explanation of embodiments of the invention. However, it will be apparent to those skilled in the art that embodiments of the invention may be practiced without these specific details. In other embodiments, well-known structures and devices are shown in block diagram form rather than in detail to avoid obscuring embodiments of the invention.

[0083] First, it's important to clarify that a data lake is typically a single storage for all of an enterprise's data. This includes copies of the raw data from the original system and transformed data generated for various tasks, such as reporting, visualization, advanced analytics, and machine learning. A data lake contains structured data (rows and columns), semi-structured data (such as CSV, logs, XML, and JSON), unstructured data (such as emails, documents, and PDFs), and binary data (such as images, audio, and video) from relational databases, forming a centralized data storage system that accommodates all forms of data.

[0084] Existing data lake processing methods, such as the one described in Chinese patent CN115576946A, involve adding index information to the metadata file of the data table, establishing a correspondence between the data file and the index in Iceberg (an open table format for massive data analysis), and then performing row-group-level index matching on the target data in Iceberg based on the correspondence between the data file and the index to determine the target data file. Subsequently, corresponding data processing operations are performed on the target row groups in the target data file to obtain the data processing results. This method only speeds up data retrieval and does not address or modify the Equality Deletes files generated during data updates. Therefore, the efficiency problem caused by Equality Deletes files remains unresolved.

[0085] The Equality deletes file is a global file that records all records with sequence numbers lower than this file. If data appears in this file, records in other data files with sequence numbers lower than this file should be deleted. This file grows larger as the data increases, causing it to consume a large amount of memory when reading data (especially when using MOR (Merge on Read) methods), and may even lead to OutOfMemoryError (OOM). At the time of data reading, the full Equality deletes file corresponding to the table snapshot is extremely large, slowing down task execution, and OOM will cause data read / write tasks to fail.

[0086] When a task writes data to a data lake table format in real time, the visibility of the data is mainly determined by the time of the snapshot submission. The data visibility in the existing data lake cannot support second-level data read and write. If second-level snapshot submission is forced, a large number of small files will be generated, which will reduce the read and write efficiency of the data lake.

[0087] Therefore, the following problems exist in current data lake read and write operations: the global Equality deletes file causes huge memory overhead, affecting data read and write efficiency; the data visibility of the existing data lake table format does not support second-level data writing; and too many small files reduce read and write efficiency.

[0088] The following explains the technical terms used in this application:

[0089] Iceberg: Iceberg is an open data lake table format for massive data analysis scenarios. A table format can be understood as a way of organizing metadata and data files, serving as an intermediate layer between the computation layer (e.g., Flink, Spark) and the storage layer (e.g., ORC, Parqurt, Avro). The table format is an abstract concept at the implementation level of a database system. Generally, a table format defines table metadata information and API (Application Programming Interface) interfaces, such as the fields included in the table, the organization of files within the table, table index information, statistical information, and the interfaces for upper-level query engines to read and write files to the table. It also includes a metadata management layer (snapshots, metadata) and a data storage layer (data files).

[0090] Master and Slave Node Services: In software architecture, the master-slave model includes a master node service (Master) and slave node services (Slave), deployed on different servers. A typical master-slave model consists of one master node service and multiple slave node services, forming a cluster. The master node service acts as a task scheduler, assigning computational tasks to the slave node services. After all slave node services have completed their tasks, the master node service aggregates the results. The master-slave model enables hot data backup, providing multiple replicas.

[0091] An equality-deletes file, also known as an equality-deletes file, must meet the following conditions when applied to a data file: the data file's sequence number is less than the deletion sequence number; the data file's partitions (normative and partition values) are both equal to the partitions of the deleted file, or the deleted file's partition norm is not partitioned. An equality-deletes file identifies deleted rows in the data file collection using one or more column values, and may optionally include other columns containing deleted rows.

[0092] The position delete file contains the following data types: the path to the data file containing the row to be deleted; the position of the row to be deleted within the data file; and the fields and their values ​​for the row to be deleted. The position delete file identifies deleted rows by their position within the file and one or more data files, and can optionally include deleted rows. If there is an entry in the position delete file that specifies the file and position of the row within a data file, that row will be deleted.

[0093] OOM, or out of memory, means that when a program requests memory, there is not enough memory available for it to use.

[0094] Snapshot: Each snapshot is a complete collection of data files in a table at a specific point in time. Snapshots are listed in the metadata file, but the files within a snapshot are stored in a separate manifest file.

[0095] WAL: Write-ahead logging. Write-ahead logging is a set of techniques used in relational database systems to provide atomicity and durability. Before any changes are committed, the system writes them to a log file managed by the WAL. The actual operations are then performed based on the log entries. This eliminates the need to wait for the system to write the changes to disk at the time of each transaction commit, as the log can be used for database recovery at any time and under any circumstances.

[0096] Memory indexes: Memory indexes are a crucial part of the database read path. For in-memory databases, memory indexes are the entry point for all queries, while for disk databases, in scenarios where the memory size is sufficient to cache most frequently accessed data, memory indexes can significantly accelerate queries for frequently accessed data.

[0097] Thread pool: A thread pool is a thread management technique that uses the pooling concept. It is mainly used to reuse threads, conveniently manage threads and tasks, and decouple thread creation from task execution. Thread pools can be created to reuse already created threads to reduce the resource consumption caused by frequent thread creation and destruction.

[0098] Data file: A file containing table rows, representing the actual table content data.

[0099] Delete files: Deletes files that encode rows deleted from a table by position or data value.

[0100] Figure 1 This is a schematic diagram illustrating the implementation environment of a data lake data processing method according to an exemplary embodiment of this application.

[0101] Reference Figure 1 As shown, the implementation environment may include a terminal device 110 and a server 120. The technical solution provided in this application embodiment can be applied to the server 120, which is used to obtain the data to be written transmitted by the terminal device 110 and to implement the writing and reading of data according to the master node service and slave node service in the server 120.

[0102] In one embodiment of this application, the slave node service in server 120 obtains the data to be written; records the data to be written in a pre-created write-ahead log file; records the data to be written in a pre-created memory partition according to the data type of the data to be written, wherein the write-ahead log file and the memory partition have a mapping relationship; generates an initial table format file corresponding to the data to be written based on the information recorded in the write-ahead log file and the memory partition, and transmits the initial table format file to the master node service to generate an initial snapshot; converts the equal-value deletion files in the initial snapshot into position deletion files, and merges the files in the initial snapshot according to the preset table format file capacity to obtain a target table format file; transmits the target table format file to the master node service to generate a target snapshot, thus completing the writing of the data to be written to the data lake. In this embodiment, the master node service and slave node service cooperate to realize the data to data lake table format processing. When writing data, the client can ignore the time when the data lake table format snapshot is generated and write the data to the slave node service in real time. Through the write-ahead log file with mapping relationship and memory partition, the data writing can be achieved at the second level, which improves the efficiency of data lake writing. The slave node service can enable data reading cache to improve the visibility of data reading in data lake table format, thereby improving the efficiency of data lake reading. The equivalent deletion files in the initial snapshot are converted into position deletion files. The data lake table format data files and position deletion files can be traversed together and processed by key to achieve the effect of data deletion. Eliminating the equivalent deletion files can eliminate the global memory index of data lake table format data, reduce memory consumption, and improve the efficiency of data lake read and write processing. According to the preset table format capacity, small files in the snapshot are merged to eliminate the number of position deletion files, which further improves the efficiency of data lake read and write processing.

[0103] It should be understood that Figure 1 The number of terminal devices 110 and servers 120 shown is merely illustrative. Any number of terminal devices 110 and servers 120 can be used as needed.

[0104] Among them, terminal device 110 corresponds to the client, which can be any electronic device with a user input interface, including but not limited to smartphones, tablets, laptops, computers, in-vehicle computers, etc. The user input interface includes but is not limited to touch screens, keyboards, physical buttons, audio pickup devices, etc.

[0105] Among them, server 120 corresponds to the server side, which can be a server that provides various services. It can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. This section does not impose any restrictions on this.

[0106] Terminal device 110 can communicate with server 120 via wireless networks such as 3G (third-generation mobile information technology), 4G (fourth-generation mobile information technology), and 5G (fifth-generation mobile information technology), and this is not restricted here.

[0107] The above sections introduced exemplary implementation environments for applying the technical solutions of this application. Next, we will continue to introduce the data lake data processing method of this application.

[0108] To address the problem of improving the efficiency of data reading and writing in data lakes in the prior art, embodiments of this application propose a data lake data processing method, a data lake data processing device, a data lake data processing system, an electronic device, a computer-readable storage medium, and a computer program product, which will be described in detail below.

[0109] Please see Figure 2 , Figure 2 This is a schematic flowchart illustrating a data lake data processing method according to an exemplary embodiment of this application. This method can be applied to… Figure 1 The implementation environment is shown. It should be understood that this method can also be applied to other exemplary implementation environments and specifically executed by devices in other implementation environments. This embodiment does not limit the implementation environment to which the method is applicable.

[0110] like Figure 2 As shown, in an exemplary embodiment, the data lake data processing method is applied to slave node services. The data lake data processing method includes at least steps S210 to S250, which are described in detail below:

[0111] In step S210, the data to be written is obtained.

[0112] For example, see Figure 3 , Figure 3This is a schematic diagram illustrating the structure of a data lake data processing system according to an exemplary embodiment of this application. The data lake data processing system includes a client (Clinent), a Master, and multiple Slaves. The Master receives the connection information from the Clinent and the heartbeat information from the Slaves, and obtains complete and usable Slave information. The Master transmits the Slave information to the Clinent, enabling the Clinent to communicate with the Slaves based on the Slave information, and to send the data to be written to the Slaves and initiate read / write requests.

[0113] In step S220, the data to be written is recorded in a pre-created write-ahead log file, and the data to be written is recorded in a pre-created memory partition according to the data type of the data to be written. There is a mapping relationship between the write-ahead log file and the memory partition.

[0114] For example, the slave creates a write-alert (WAL) and a memory partition (also known as a memory slice) based on a preset file size. After receiving data to be written, the slave records the data in the WAL and then records the data in the memory partition according to the data type.

[0115] In step S230, based on the information recorded in the write-ahead log file and the memory partition, an initial table format file corresponding to the data to be written is generated, and the initial table format file is transmitted to the master node service to generate an initial snapshot.

[0116] For example, the Slave consumes a pre-defined tabular file generation thread, periodically traversing the WAL records and writing them to Iceberg. During this process, the WAL and the memory partition work together to write data to Iceberg. The Slave consumes the pre-defined tabular file generation thread, writing data from the WAL to the data-file and position-file respectively based on the Delete and Update records in the memory partition, fully adhering to the Iceberg data file structure, thus completing the WAL to Iceberg conversion. After the conversion, the Iceberg file is transmitted to the Master for initial snapshot generation. Upon receiving the Iceberg file from the Slave, the Master begins the snapshot submission task. After successful snapshot submission, the Master sends a notification to the Slave. The Slave then deletes the data from the corresponding WAL and memory partition based on the notification.

[0117] In step S240, the equivalent deletion files in the initial snapshot are converted into location deletion files, and the files in the initial snapshot are merged according to the preset tabular file size to obtain the target tabular file.

[0118] For example, the threads in the Slave's asynchronous optimization thread pool periodically execute Iceberg's table optimization tasks, converting existing Equality deletes files into Position Delete Files; merging a large number of small data files and their corresponding Position Delete Files to generate files of appropriate size; and cleaning up expired snapshots.

[0119] In step S250, the target table file is transferred to the master node service to generate a target snapshot, thus completing the writing of the data to be written to the data lake.

[0120] As can be seen from steps S210 to S250 above, the scheme proposed in this embodiment uses the master node service and the slave node service to cooperate in processing data into a data lake table format. When writing data, the client can ignore the time when the data lake table format snapshot is generated and write the data to the slave node service in real time. Through the pre-written log file with a mapping relationship and memory partition, data writing can be achieved at the second level, improving the efficiency of writing data to the data lake. The slave node service can enable caching for reading data, improving the visibility of reading data in the data lake table format, thereby improving the efficiency of reading data to the data lake. The equivalent deletion files in the initial snapshot are converted into position deletion files. The data lake table format data files and position deletion files can be traversed together and processed by key to achieve the effect of deleting data. Eliminating the equivalent deletion files can eliminate the global memory index of the data lake table format data, reduce memory consumption, and improve the efficiency of data lake read and write processing. According to the preset table format capacity, small files in the snapshot are merged to eliminate the number of position deletion files, further improving the efficiency of data lake read and write processing.

[0121] In one embodiment of this application, see Figure 4 , Figure 4 This is a schematic diagram illustrating the structure of a client in an exemplary embodiment of this application. Figure 4 As shown, the client includes a Master connector for connecting to the Master to obtain the Slave list; a timed discoverer for periodically communicating with the master to obtain Slave information and updating the Slave list; and a Slave connector for connecting to and communicating with Slaves based on the Slave list, sending data write commands and data read commands.

[0122] In one embodiment of this application, see Figure 5 and Figure 6 , Figure 5 This is a schematic diagram illustrating the structure of the master node service in an exemplary embodiment of this application. Figure 6 This is a schematic diagram illustrating the internal interactions of the master node service, as shown in an exemplary embodiment of this application. Figure 5 and Figure 6 As shown, the master node services include: Slave heartbeat receiver, Slave list, asynchronous processing thread pool, Iceberg snapshot processor, client connector, and Slave broadcaster.

[0123] Among them, such as Figure 6 As shown:

[0124] The Slave heartbeat receiver is used to receive heartbeat information and Iceberg files sent by the Slave.

[0125] The Slave list is used to record all relevant information about the Slave, including basic Slave information (heartbeat information) and the Slave's file list (file information for WAL-converted Iceberg).

[0126] The threads in the asynchronous processing thread pool mainly perform two types of tasks: generating Iceberg table optimization tasks and sending them to the broadcaster; and generating Iceberg snapshot tasks and executing them.

[0127] The Iceberg snapshot processor primarily works by retrieving the Data File and Position Delete File information already generated by the Slave from the Slave file list when an Iceberg snapshot task begins execution. It then generates a new snapshot ID, creates a new Iceberg snapshot, and submits the generated snapshot to Iceberg. Upon successful snapshot submission, the results are transmitted to the Slave broadcaster. The submission results include the Slave ID and a list of Data File and Position Delete File paths.

[0128] The Slave broadcaster primarily broadcasts snapshot commit results and table optimization tasks.

[0129] The client connector primarily accepts client connections and responds with available Slave information. Once the client receives the Slave information, it can communicate with the Slave and initiate read / write data requests.

[0130] In one embodiment of this application, see Figure 7 and Figure 8 , Figure 7 This is a schematic diagram illustrating the structure of a slave node service, as shown in an exemplary embodiment of this application. Figure 8 This is a schematic diagram illustrating the internal interaction of a slave node service, as shown in an exemplary embodiment of this application. Figure 7As shown, the slave node services include: heartbeat reporter, write-ahead log processor (WAL), write-ahead log queue, memory cache partition processor (mem segment), asynchronous consumer thread pool, asynchronous optimization thread pool, Iceberg file generator, Iceberg reader, and cleaner.

[0131] Among them, such as Figure 8 As shown:

[0132] The main tasks of the heartbeat reporter include: establishing a connection with the master; periodically sending its own heartbeat information to the master; receiving the results of the Iceberg file generator and sending them to the master in a timely manner; and receiving Iceberg optimization tasks in the response information and adding them to the optimization task queue.

[0133] The write-ahead log processor and the memory cache partition processor work together to record data to be written.

[0134] The asynchronous consumer thread pool uses threads to periodically read files from the queue to convert WAL files into Iceberg files.

[0135] The asynchronous optimization thread pool periodically reads the optimization task list and executes the optimization tasks, mainly performing the following optimization tasks: merging small files; clearing expired snapshots; rewriting the manifest; clearing orphaned files; and converting EqualityDeletesFile to Position DeletesFile.

[0136] The cleaner primarily receives snapshot submission results from Iceberg broadcast by the Master and deletes data from the WAL and memory partitions based on these results.

[0137] The Iceberg reader receives read data requests from clients, retrieves the data according to the read requirements, and merges the read results with the current mem segment.

[0138] In one embodiment of this application, before obtaining the data to be written, the method further includes:

[0139] The heartbeat information of the slave node service is sent to the master node service, so that the master node service generates a list of available slave node services based on the heartbeat information.

[0140] The master node service transmits the list of available slave node services to the client, so that the client can select available slave node services according to the list and issue data processing instructions.

[0141] In one embodiment of this application, selecting an available slave node service based on the available slave node service list and issuing a data processing instruction includes:

[0142] Select multiple available slave node services from the list of available slave node services, perform load balancing processing based on preset Internet Protocol information and the multiple available slave node services, and establish communication connections with the multiple available slave node services;

[0143] Data write instructions and / or data read instructions are sent to each available slave node service in the communication connection to realize data processing of the data lake.

[0144] In one embodiment of this application, the heartbeat information includes the slave node service name, slave node service identity identifier, slave node service Internet Protocol address, and slave node service port information;

[0145] The target table format file includes data file path information, data file serial number, location-deleted file path information, location-deleted file serial number, write-ahead log file, and slave node service identity identifier.

[0146] For example, heartbeat information includes the following:

[0147] Name: The name of the slave;

[0148] ID: The Slave's ID, a unique identifier within this service;

[0149] IP: The Slave's IP address, used for communication;

[0150] PORT: The slave's port, used for communication.

[0151] The Iceberg file includes:

[0152] Data File Path: The path to the data file;

[0153] Data File number: The sequence number of the data file;

[0154] Position Delete File Path: Position the path of the file to be deleted;

[0155] Position Delete File number: The serial number of the file to be deleted at that position;

[0156] WAL File: Write-Ahead Log File;

[0157] Slave ID: The ID of the slave node service.

[0158] In one embodiment of this application, recording the data to be written in a pre-created memory partition according to the data type of the data to be written includes:

[0159] When the data type is data to be deleted, the data to be written is recorded in the deletes partition, and the records corresponding to the data to be written in the indexes partition and the List partition are deleted.

[0160] When the data type is update data and the indexes partition contains a record corresponding to the data to be written, the record corresponding to the data to be written in the indexes partition is updated to the List partition;

[0161] When the data type is update data and there is no record corresponding to the data to be written in the indexes partition, the data to be written is recorded in the deletes partition and the list partition, and the record position of the data to be written in the list partition is recorded in the indexes partition;

[0162] When the data type is inserted data, the data to be written is recorded in the List partition;

[0163] The deletes partition, indexes partition, and list partition are included in the pre-created memory partition.

[0164] For example, see Figure 9 , Figure 9 This is a schematic diagram illustrating the write-ahead log sharding and memory partitioning structure, as shown in an exemplary embodiment of this application. The write-ahead log shards and memory partitions work together to perform the following tasks:

[0165] Open a write-ahead log file and pre-allocate a certain amount of space for it;

[0166] Open read / write channels for the write-ahead log file using memory mapping technology;

[0167] Allocate a contiguous block of memory for caching purposes, and divide this space into a List (a list of data), indexes ((Map[key, the index of the data in the List])), and deletes (the key of the record to be deleted);

[0168] When a write data request is received, the data is recorded in the WAL and flushed to the disk.

[0169] For example, see Figure 10 , Figure 10 This is an exemplary embodiment of the present application illustrating the recording of data to be written to a write-ahead log file and a memory partition. For example... Figure 10 As shown, recording the data to be written to the write-ahead log file and the memory partition includes the following steps:

[0170] If the data type is Delete, add it to deletes and remove the record from indexes. If the record to be removed exists in indexes, get the index and delete the data at the corresponding index in the list.

[0171] If this data is an update, first check if it exists in the index. If it does not exist, add the record in deletes, add the record at the end of the list, and add a record in indexes [data, index at the end of the list]. If a record exists in the index, use the position in the list indicated by this record to directly update the data in the list.

[0172] If the data is inserted, add this record to the end of the list.

[0173] In one embodiment of this application, when the write-ahead log file is full of data, the write-ahead log file is closed and added to a pre-built write-ahead log file queue;

[0174] When the memory partition is full of data, the write-ahead log file is closed and added to a pre-built write-ahead log file queue.

[0175] If the write-ahead log file queue is empty when the slave node service consumes the preset table format file generation thread, then the write-ahead log file with currently existing data is closed, and the write-ahead log file is added to the write-ahead log file queue.

[0176] In one embodiment of this application, generating an initial table format file corresponding to the data to be written, based on the write-ahead log file and the information recorded in the memory partition, includes:

[0177] When the data type is inserted data, the data to be written is transferred to the target data lake table format data file;

[0178] When the data type is data to be deleted, the data to be written is transferred to the equivalent deletion file of the target data lake table format;

[0179] When the data type is update data and it is recorded in the deletes partition, the data to be written is transferred to the data file, the position information of the data to be written in the data file is recorded, and the data to be written is transferred to the equivalent deletion file;

[0180] When the data type is update data and it is recorded in the indexes partition, the data to be written is transferred to the data file, the position information of the data to be written in the data file is recorded, and the position information is recorded in the position deletion file of the target data lake table format.

[0181] For example, see Figure 11 , Figure 11 This is a schematic diagram illustrating the generation of a tabular file, as shown in an exemplary embodiment of this application. Figure 11 As shown, generating a tabular file involves the following steps:

[0182] S1101, Traverse the data in the WAL file.

[0183] S1102. If the data type is insert, write the data directly to the Data File.

[0184] S1103. If the data type is Delete, then it is recorded in the Equality deletes File.

[0185] S1104. If the data type is Updat, write the data to the Data File and record the current position; if it also exists in the cached deletes, record it in the Equality deletes File; if it also exists in the cached indexes, record this data in the Position Deletes File [data, Data File Path, Data File position].

[0186] S1105. Traverse the Equality deletes file generated in step S1104 and convert it into a PositionDeletes file.

[0187] S1106. Send the result to the heartbeat reporter.

[0188] In one embodiment of this application, converting the equivalent deletion file in the initial snapshot into a location deletion file includes:

[0189] Based on the sequence number of the equally deleted files in the initial snapshot, the equally deleted files are sorted in ascending order;

[0190] Traverse the equivalent deletion files in ascending order and store the records in the equivalent deletion files into the memory cache;

[0191] Obtain the files to be converted from the data file whose serial numbers are less than the serial numbers of the files to be deleted;

[0192] When the file to be converted exists in the memory cache, a new location deletion file is generated based on the file to be converted, and the serial number of the equivalent deletion file, the path of the data file, and the position of the file to be converted in the data file are added to the new location deletion file.

[0193] For example, generating a tabular file includes the following steps:

[0194] a. Retrieve the Equality deletes file under the current snapshot and sort it in ascending order by sequence number.

[0195] b. Traverse the sorted Equality deletes file, read the records of the current Equality deletes file into the memory cache. If it is too large, it can be read in segments and traversed in each segment.

[0196] c. Iterate through the records in the Data File whose Sequence number is less than the Sequence number of the Equality deletes File in step b, and denote them as records. If the record exists in the cache in step b, generate a Position Delete File with its Sequence number being the Sequence number of the current Data File (if the file exists, it will not be created again). Add a deletion record to the Position Delete File: Equality ids, the path of the current Data File, and the position of the record in the Data File.

[0197] d. After traversing all n Equality deletes files, generate an Iceberg snapshot.

[0198] e. Repeat steps a to d to ensure that there are no Equality deletes files in the Iceberg table.

[0199] In one embodiment of this application, merging files in the initial snapshot according to a preset tabular file size includes:

[0200] Obtain the data files to be merged and the files to be deleted at the merge locations from the initial snapshot whose file size is smaller than the preset first table format file size;

[0201] According to the preset second table format file capacity, multiple data files to be merged are merged, and multiple files to be deleted at the merged location are merged, wherein the capacity of the second table format file is greater than the capacity of the first table format file.

[0202] In one embodiment of this application, after transmitting the target tabular file to the master node service to generate a target snapshot, the method further includes:

[0203] When receiving the target snapshot submission result transmitted by the master node service, the name of the write-ahead log file to be deleted is obtained from the target snapshot submission result.

[0204] Based on the name of the write-ahead log file to be deleted, delete the write-ahead log file in the write-ahead log file queue and delete the data in the memory partition corresponding to the name of the write-ahead log file to be deleted.

[0205] For example, the cleaner deletes log files with the specified WAL names in the results and marks the corresponding memory caches of the above WALs as eligible for expiration and cleanup. The cleaner also periodically cleans up cached data marked as expired, based on the rule that its time to life has exceeded a specified period.

[0206] In one embodiment of this application, after writing the data to be written to the data lake is completed, the method further includes:

[0207] In response to a data read command, the read cache function of the target data lake table format is enabled to obtain the target data in the target data lake table format;

[0208] Delete the record corresponding to the target data in the deletes partition, and update the record corresponding to the target data in the List partition to complete the reading of the target data in the data lake.

[0209] For example, the Iceberg reader receives a read data request from a client, retrieves the data according to the read requirements, and merges the read result with the current mem segment. The merging steps are as follows: if a record exists in `deletes`, it is deleted; records in the `List` partition overwrite the queried records. It's important to note that the Iceberg reader enables read caching to accelerate data reading efficiency.

[0210] When a request to read data is made, the Iceberg read caching feature on the Slave is enabled to cache frequently accessed data. After retrieving the data from Iceberg, it is then purged from delete records in the Slave's memory partition and merged with data records to obtain the final data. Since both the memory partition and Iceberg's read caching reside in memory, data is read from Iceberg starting with the latest snapshot, achieving read response times in the order of seconds.

[0211] In one embodiment of this application, when the broadcaster receives the snapshot submission result, it iterates through the Slave list and sends the result to each Slave one by one. When sending, it needs to filter according to the Slave ID and only send the submission result of the corresponding Slave.

[0212] When the broadcaster receives an optimization task, it starts the task allocator, iterates through Iceberg's tables, and binds the optimization task to the table using a round-robin approach before sending it to the slave. This ensures that each slave receives a certain number of optimization tasks for each table. Each time a table is polled and tasks are sent to a slave, the slave list is randomly sorted to ensure that optimization tasks for each table have a chance to be executed on every slave, preventing any single slave from being under constant heavy load.

[0213] In the above embodiment, through the cooperation of Master and Slave, the client does not need to worry about the snapshot generation time of Iceberg and can write data to Slave in real time, indicating that the write has been successful. The WAL in Slave accelerates the data write operation through memory mapping, achieving write speeds down to the second level. Since Iceberg on the Slave side has enabled read data caching, the visibility of Iceberg read data is greatly improved. In addition, newly added data that has not been persisted to Iceberg is in the Slave's cache and can be quickly retrieved and merged with the data read from Iceberg, achieving the final effect of fast data reading. As can be seen from the implementation of the table optimization task and WAL to Iceberg task on the Slave side, the potential Equality Deletes File is converted into a Position Deletes File. Combined with the Iceberg data reading logic, the Data File and Position Deletes File can be traversed together and processed by key group to achieve the effect of deleting data. Eliminating the Equality Deletes File naturally eliminates the global memory index of the table data. Eliminating the index naturally reduces memory consumption and avoids the occurrence of OOM (Out of Memory). Regularly merging files further eliminates Position Deletes files, which in turn accelerates data reading speed.

[0214] In one embodiment of this application, the data lake data processing method of this application is actually verified. The verification results show that the data lake data processing method can accurately control the memory overhead required by the service, optimize client memory by more than 90%, improve Iceberg table read efficiency by 400 times, and improve Iceberg table write efficiency by 60 times.

[0215] Figure 12 This is a block diagram illustrating a data lake data processing apparatus according to an exemplary embodiment of this application. The apparatus can be applied to… Figure 1 The implementation environment shown is not limited to this embodiment. This device can also be applied to other exemplary implementation environments and specifically configured in other devices. This embodiment does not limit the implementation environment to which the device is applicable.

[0216] like Figure 12 As shown, the exemplary data lake data processing apparatus includes:

[0217] Data acquisition module 1201 is used to acquire data to be written;

[0218] The data recording module 1202 is used to record the data to be written in a pre-created write-ahead log file, and to record the data to be written in a pre-created memory partition according to the data type of the data to be written, wherein there is a mapping relationship between the write-ahead log file and the memory partition;

[0219] The initial snapshot generation module 1203 is used to generate an initial table format file corresponding to the data to be written based on the write-ahead log file and the information recorded in the memory partition, and to transmit the initial table format file to the master node service to generate an initial snapshot;

[0220] The optimization module 1204 is used to convert the equivalent deletion files in the initial snapshot into position deletion files, and merge the files in the initial snapshot according to the preset tabular file size to obtain the target tabular file;

[0221] The target snapshot generation module 1205 is used to transmit the target tabular file to the master node service to generate a target snapshot, thereby completing the writing of the data to be written into the data lake.

[0222] In this exemplary data lake data processing device, the master node service and the slave node service cooperate to realize the data to data lake table format processing. When writing data, the client can ignore the time when the data lake table format snapshot is generated and write the data to the slave node service in real time. Through the write-ahead log file with a mapping relationship and memory partition, the data writing can be achieved at the second level, which improves the efficiency of data lake writing. The slave node service can enable data reading cache to improve the visibility of data reading in data lake table format, thereby improving the efficiency of data lake reading. The equivalent deletion files in the initial snapshot are converted into position deletion files. The data lake table format data files and position deletion files can be traversed together and processed by key to achieve the effect of data deletion. Eliminating the equivalent deletion files can eliminate the global memory index of data lake table format data, reduce memory consumption, and improve the efficiency of data lake read and write processing. According to the preset table format capacity, small files in the snapshot are merged to eliminate the number of position deletion files, which further improves the efficiency of data lake read and write processing.

[0223] It should be noted that the data lake data processing apparatus and the data lake data processing method provided in the above embodiments belong to the same concept. The specific ways in which each module and unit performs operations have been described in detail in the method embodiments and will not be repeated here. In practical applications, the data lake data processing apparatus provided in the above embodiments can be assigned to different functional modules as needed, that is, the internal structure of the apparatus can be divided into different functional modules to complete all or part of the functions described above. This is not a limitation here.

[0224] In one embodiment of this application, see Figure 3 The data lake data processing system includes:

[0225] The slave node service is used to obtain the data to be written; record the data to be written in a pre-created write-ahead log file; record the data to be written in a pre-created memory partition according to the data type of the data to be written, wherein the write-ahead log file and the memory partition have a mapping relationship; generate an initial table format file corresponding to the data to be written based on the information recorded in the write-ahead log file and the memory partition, and transmit the initial table format file to the master node service to generate an initial snapshot; convert the equal-value deletion files in the initial snapshot into position deletion files, and merge the files in the initial snapshot according to the preset table format file capacity to obtain a target table format file; transmit the target table format file to the master node service to generate a target snapshot, thus completing the writing of the data to be written to the data lake;

[0226] The master node service is used to receive the initial tabular file to generate an initial snapshot, and to receive the target tabular file to generate a target snapshot;

[0227] The client is used to transmit the data to be written to the slave node service.

[0228] It should be noted that the data lake data processing system provided in the above embodiments and the data lake data processing method provided in the above embodiments belong to the same concept. The working methods of the slave node service, master node service and client have been described in detail in the method embodiments, and will not be repeated here.

[0229] Embodiments of this application also provide an electronic device, including: one or more processors; and a storage device for storing one or more programs, which, when executed by the one or more processors, cause the electronic device to implement the data lake data processing method provided in the above embodiments.

[0230] Figure 13 A schematic diagram of a computer system suitable for an electronic device according to an embodiment of this application is shown. It should be noted that... Figure 13 The computer system 1300 of the electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0231] like Figure 13As shown, the computer system 1300 includes a Central Processing Unit (CPU) 1301, which can perform various appropriate actions and processes based on programs stored in Read-Only Memory (ROM) 1302 or programs loaded from Storage Unit 1308 into Random Access Memory (RAM) 1303, such as performing the methods described in the above embodiments. Various programs and data required for system operation are also stored in RAM 1303. The CPU 1301, ROM 1302, and RAM 1303 are interconnected via bus 1304. An Input / Output (I / O) interface 1305 is also connected to bus 1304.

[0232] The following components are connected to I / O interface 1305: an input section 1306 including a keyboard, mouse, etc.; an output section 1307 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 1308 including a hard disk, etc.; and a communication section 1309 including a network interface card such as a LAN (Local Area Network) card, modem, etc. The communication section 1309 performs communication processing via a network such as the Internet. A drive 1310 is also connected to I / O interface 1305 as needed. Removable media 1311, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., are installed on drive 1310 as needed so that computer programs read from them can be installed into storage section 1308 as needed.

[0233] Specifically, according to embodiments of this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program including a computer program for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 1309, and / or installed from removable medium 1311. When the computer program is executed by central processing unit (CPU) 1301, it performs various functions defined in the system of this application.

[0234] It should be noted that the computer-readable medium shown in the embodiments of this application can be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. A computer-readable storage medium can be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, optical fiber, portable compact disc read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this application, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying a computer-readable computer program. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The computer program contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to wireless, wired, etc., or any suitable combination thereof.

[0235] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. Each block in a flowchart or block diagram may represent a module, segment, or portion of code, which contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0236] The units described in the embodiments of this application can be implemented in software or hardware, and the described units can also be located in a processor. The names of these units do not necessarily limit the specific unit itself.

[0237] Another aspect of this application provides a computer-readable storage medium storing a computer program thereon, which, when executed by a computer's processor, causes the computer to perform the data lake data processing method as described above. This computer-readable storage medium may be included in the electronic device described in the above embodiments, or it may exist independently and not assembled into the electronic device.

[0238] Another aspect of this application provides a computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the data lake data processing method provided in the various embodiments described above.

[0239] The above embodiments are merely illustrative of the principles and effects of the present invention and are not intended to limit the invention. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in the present invention should still be covered by the claims of the present invention.

Claims

1. A data lake data processing method, characterized in that, The method is applied to a slave node service, and the method includes: Get the data to be written; The data to be written is recorded in a pre-created write-ahead log file, and the data to be written is also recorded in a pre-created memory partition according to the data type of the data to be written, wherein there is a mapping relationship between the write-ahead log file and the memory partition; Based on the information recorded in the write-ahead log file and the memory partition, an initial table format file corresponding to the data to be written is generated, and the initial table format file is transmitted to the master node service to generate an initial snapshot; The files in the initial snapshot that are deleted by equal value are converted into files that are deleted by location, and the files in the initial snapshot are merged according to the preset tabular file size to obtain the target tabular file; The target table file is transferred to the master node service to generate a target snapshot, thus completing the writing of the data to be written to the data lake.

2. The data lake data processing method according to claim 1, characterized in that, Before obtaining the data to be written, the following steps are also included: The heartbeat information of the slave node service is sent to the master node service, so that the master node service generates a list of available slave node services based on the heartbeat information. The master node service transmits the list of available slave node services to the client, so that the client can select available slave node services according to the list and issue data processing instructions.

3. The data lake data processing method according to claim 2, characterized in that, Select available slave node services from the list of available slave node services and issue data processing instructions, including: Select multiple available slave node services from the list of available slave node services, perform load balancing processing based on preset Internet Protocol information and the multiple available slave node services, and establish communication connections with the multiple available slave node services; Data write instructions and / or data read instructions are sent to each available slave node service in the communication connection to realize data processing of the data lake.

4. The data lake data processing method according to claim 2, characterized in that: The heartbeat information includes the slave node service name, slave node service identity identifier, slave node service Internet Protocol address, and slave node service port information; The target table format file includes data file path information, data file serial number, location-deleted file path information, location-deleted file serial number, write-ahead log file, and slave node service identity identifier.

5. The data lake data processing method according to claim 4, characterized in that, Recording the data to be written in a pre-created memory partition according to the data type of the data to be written includes: When the data type is data to be deleted, the data to be written is recorded in the deletes partition, and the records corresponding to the data to be written in the indexes partition and the List partition are deleted. When the data type is update data and the indexes partition contains a record corresponding to the data to be written, the record corresponding to the data to be written in the indexes partition is updated to the List partition; When the data type is update data and there is no record corresponding to the data to be written in the indexes partition, the data to be written is recorded in the deletes partition and the list partition, and the record position of the data to be written in the list partition is recorded in the indexes partition; When the data type is inserted data, the data to be written is recorded in the List partition; The deletes partition, indexes partition, and list partition are included in the pre-created memory partition.

6. The data lake data processing method according to claim 5, characterized in that: When the write-ahead log file is full of data, the write-ahead log file is closed and added to a pre-built write-ahead log file queue; When the memory partition is full of data, the write-ahead log file is closed and added to a pre-built write-ahead log file queue. If the write-ahead log file queue is empty when the slave node service consumes the preset table format file generation thread, then the write-ahead log file with currently existing data is closed, and the write-ahead log file is added to the write-ahead log file queue.

7. The data lake data processing method according to claim 5, characterized in that, Based on the write-ahead log file and the information recorded in the memory partition, an initial table format file corresponding to the data to be written is generated, including: When the data type is inserted data, the data to be written is transferred to the target data lake table format data file; When the data type is data to be deleted, the data to be written is transferred to the equivalent deletion file of the target data lake table format; When the data type is update data and it is recorded in the deletes partition, the data to be written is transferred to the data file, the position information of the data to be written in the data file is recorded, and the data to be written is transferred to the equivalent deletion file; When the data type is update data and it is recorded in the indexes partition, the data to be written is transferred to the data file, the position information of the data to be written in the data file is recorded, and the position information is recorded in the position deletion file of the target data lake table format.

8. The data lake data processing method according to claim 7, characterized in that, Converting the equal-value deleted files in the initial snapshot into location-deleted files includes: Based on the sequence number of the equally deleted files in the initial snapshot, the equally deleted files are sorted in ascending order; Traverse the equivalent deletion files in ascending order and store the records in the equivalent deletion files into the memory cache; Obtain the files to be converted from the data file whose serial numbers are less than the serial numbers of the files to be deleted; When the file to be converted exists in the memory cache, a new location deletion file is generated based on the file to be converted, and the serial number of the equivalent deletion file, the path of the data file, and the position of the file to be converted in the data file are added to the new location deletion file.

9. The data lake data processing method according to claim 1, characterized in that, Merging files from the initial snapshot according to the preset tabular file size includes: Obtain the data files to be merged and the files to be deleted at the merge locations from the initial snapshot whose file size is smaller than the preset first table format file size; According to the preset second table format file capacity, multiple data files to be merged are merged, and multiple files to be deleted at the merged location are merged, wherein the capacity of the second table format file is greater than the capacity of the first table format file.

10. The data lake data processing method according to claim 6, characterized in that, After transferring the target table file to the master node service to generate the target snapshot, the process also includes: When receiving the target snapshot submission result transmitted by the master node service, the name of the write-ahead log file to be deleted is obtained from the target snapshot submission result; Based on the name of the write-ahead log file to be deleted, delete the write-ahead log file in the write-ahead log file queue and delete the data in the memory partition corresponding to the name of the write-ahead log file to be deleted.

11. The data lake data processing method according to claim 7, characterized in that, After writing the data to be written to the data lake is completed, the process also includes: In response to a data read command, the read cache function of the target data lake table format is enabled to obtain the target data in the target data lake table format; Delete the record corresponding to the target data in the deletes partition, and update the record corresponding to the target data in the List partition to complete the reading of the target data in the data lake.

12. A data lake data processing device, characterized in that, The device includes: The data acquisition module is used to acquire the data to be written. The data recording module is used to record the data to be written in a pre-created write-ahead log file, and to record the data to be written in a pre-created memory partition according to the data type of the data to be written, wherein there is a mapping relationship between the write-ahead log file and the memory partition; The initial snapshot generation module is used to generate an initial table format file corresponding to the data to be written based on the write-ahead log file and the information recorded in the memory partition, and to transmit the initial table format file to the master node service to generate an initial snapshot; The optimization module is used to convert the files with equal deletion value in the initial snapshot into files with location deletion value, and merge the files in the initial snapshot according to the preset tabular file size to obtain the target tabular file; The target snapshot generation module is used to transfer the target tabular file to the master node service to generate a target snapshot, thereby completing the writing of the data to be written into the data lake.

13. A data lake data processing system, characterized in that, The system includes: The slave node service is used to obtain the data to be written; record the data to be written in a pre-created write-ahead log file; record the data to be written in a pre-created memory partition according to the data type of the data to be written, wherein there is a mapping relationship between the write-ahead log file and the memory partition; generate an initial table format file corresponding to the data to be written based on the information recorded in the write-ahead log file and the memory partition, and transmit the initial table format file to the master node service to generate an initial snapshot; convert the equal-value deletion files in the initial snapshot into position deletion files, and merge the files in the initial snapshot according to the preset table format file capacity to obtain a target table format file; transmit the target table format file to the master node service to generate a target snapshot, thus completing the writing of the data to be written to the data lake; The master node service is used to receive the initial tabular file to generate an initial snapshot, and to receive the target tabular file to generate a target snapshot; The client is used to transmit the data to be written to the slave node service.

14. An electronic device, characterized in that, The electronic device includes: One or more processors; A storage device for storing one or more programs, which, when executed by one or more processors, cause the electronic device to implement the data lake data processing method as described in any one of claims 1 to 11.

15. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed by the computer's processor, causes the computer to perform the data lake data processing method as described in any one of claims 1 to 11.

Images