Data into lake method, device, equipment, medium and program product

Abstract

The invention provides a data lake entering method, and relates to the technical field of big data. The method comprises the steps of obtaining a target table meeting application conditions of a target generation mode; generating a target lake entering file of the target table according to a processing flow of the target generation mode; importing the data of the target table into the data lake through the target lake entering file; wherein the target generation mode is one of a single-fragment generation mode, a multi-fragment generation mode or an asynchronous generation mode; wherein the application condition of the single fragment generation mode comprises that the storage mode of the table is a single fragment; the application conditions of the asynchronous generation mode include that the storage mode of the table is a multi-fragment mode, the data volume of at least one first fragment is greater than a threshold data volume, and a preset field exists in the table; the application condition of the multi-fragment generation mode comprises that the storage mode of the table is multi-fragment but does not meet the application condition of the asynchronous generation mode. The invention further provides a data lake entering device and equipment, a storage medium and a program product.

Description

Technical Field

[0001] This invention relates to the field of big data, and more specifically to a method, apparatus, device, medium, and program product for data import into a lake. Background Technology

[0002] Data ingestion into a big data platform is the process of migrating data from various data sources to the platform, and it forms the foundation for big data processing and analysis. The process involves generating ingestion files for the data tables from various data sources, and then loading the corresponding data into the big data platform using these files. These data sources involve numerous tables, which often differ significantly in table structure, storage methods, and the available timeframes for data ingestion. This often necessitates analyzing and processing each table individually during the ingestion process, resulting in a substantial workload and high time and resource costs. Summary of the Invention

[0003] In view of the above problems, the present invention provides a general method, apparatus, device, medium and program product for batch processing of data entering a lake.

[0004] A first aspect of this invention provides a data import method. The method includes: obtaining a target table that meets the applicable conditions of a target generation method; generating a target import file for the target table according to the processing flow of the target generation method; and importing the data of the target table into a data lake through the target import file. The target generation method is one of a single-shard generation method, a multi-shard generation method, or an asynchronous generation method. The applicable conditions for the single-shard generation method include that the table is stored in a single-shard configuration; the applicable conditions for the asynchronous generation method include that the table is stored in a multi-shard configuration, and the data volume of at least one first shard is greater than a threshold data volume, and a preset field exists in the table; the applicable conditions for the multi-shard generation method include that the table is stored in a multi-shard configuration, but the applicable conditions for the asynchronous generation method are not met. The single-shard generation method processes data on a single shard as a unit; the multi-shard generation method processes data on each shard as a unit; and the asynchronous generation method processes data on the first shard in units of data split according to the preset field.

[0005] According to an embodiment of the present invention, the operations of obtaining the target table and generating the target entry file are performed by a target batch job corresponding to the target generation method; wherein, for the single-shard generation method, the multi-shard generation method and the asynchronous generation method, respectively, there are corresponding batch jobs configured.

[0006] According to an embodiment of the present invention, obtaining the target table that satisfies the applicable conditions of the target generation method includes: reading the target entry configuration file through the target batch job to obtain the target table, wherein the target entry configuration file is configured with the table name of the target table that satisfies the applicable conditions of the target generation method.

[0007] According to an embodiment of the present invention, the target entry configuration file is further configured with the execution time information of the target table, wherein the target batch job generates the target entry file within the time range specified by the execution time information of the target table.

[0008] According to an embodiment of the present invention, the method further includes: starting batch jobs corresponding to the single-shard generation method, multi-shard generation method, and asynchronous generation method in each session at the beginning of each session. The execution time information of the target table includes information about the sessions in which the target table is processed, wherein the target table is processed within the sessions specified by the execution time information of the target table to generate the target lake entry file.

[0009] According to an embodiment of the present invention, before obtaining the target table that satisfies the applicable conditions of the target generation method, the method further includes: updating the target entry configuration file in response to a configuration update operation that adds a new table to the target entry configuration file.

[0010] According to an embodiment of the present invention, generating the target entry file of the target table according to the processing flow of the target generation method includes: obtaining the target query configuration information of the target table; reading data from the target table according to the query statement configured in the target query configuration information; and generating the target entry file based on the read data.

[0011] According to an embodiment of the present invention, when the target processing method is a multi-shard generation method, the step of generating the target entry file of the target table according to the processing flow of the target generation method includes: generating a sub-entry file for each shard of the target table; and merging the sub-entry files of each shard of the target table into the target entry file.

[0012] According to an embodiment of the present invention, when the target processing method is an asynchronous generation method, generating the target lake entry file of the target table according to the processing flow of the target generation method includes: generating a sub-lake entry file for each shard of the target table; and merging the sub-lake entry files of each shard of the target table into the target lake entry file. The step of generating a sub-lake entry file for each shard of the target table includes: splitting the data in the first shard into n parts according to the preset field, where n is an integer greater than 1; generating a local sub-file for each of the n parts; and merging the local sub-files corresponding to the n parts to obtain the sub-lake entry file of the first shard.

[0013] A second aspect of this invention provides a data import device. The device includes an acquisition module, a generation module, and an import module.

[0014] The acquisition module is used to acquire target tables that meet the applicable conditions of the target generation method. The target generation method is one of a single-shard generation method, a multi-shard generation method, or an asynchronous generation method. The applicable conditions for the single-shard generation method include that the table is stored in a single-shard configuration. The applicable conditions for the asynchronous generation method include that the table is stored in a multi-shard configuration, and the data volume of at least one first shard is greater than a threshold data volume, and the table contains a preset field. The applicable conditions for the multi-shard generation method include that the table is stored in a multi-shard configuration, but the applicable conditions for the asynchronous generation method are not met.

[0015] The generation module is used to generate the target entry file of the target table according to the processing flow of the target generation method. Specifically, the single-shard generation method processes data on a single shard basis, the multi-shard generation method processes data on each shard basis, and the asynchronous generation method processes the data on the first shard in units of data split according to the preset field.

[0016] The import module is used to import data from the target table into the data lake using the target import file.

[0017] According to an embodiment of the present invention, the apparatus further includes a first configuration module. The first configuration module is configured with corresponding batch jobs for single-shard generation, multi-shard generation, and asynchronous generation methods, respectively. The acquisition module is used to acquire a target table that meets the applicable conditions of the target generation method through the target batch job corresponding to the target generation method. The generation module is used to generate the target lake entry file of the target table by executing the processing flow of the target generation method through the target batch job corresponding to the target generation method.

[0018] According to an embodiment of the present invention, the apparatus further includes a second configuration module. The second configuration module is used to configure a target entry configuration file, wherein the target entry configuration file contains table names of the target tables that satisfy the applicable conditions of the target generation method. The acquisition module is specifically used to read the target entry configuration file through the target batch job to obtain the target tables.

[0019] A third aspect of the present invention provides an electronic device comprising: one or more processors; and a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the method described above.

[0020] A fourth aspect of the present invention also provides a computer-readable storage medium having a computer program or instructions stored thereon, wherein the computer program or instructions, when executed by a processor, implement the steps of the above-described method.

[0021] A fifth aspect of the present invention also provides a computer program product, including a computer program or instructions that, when executed by a processor, implement the steps of the above-described method. Attached Figure Description

[0022] The above-described features, other objects, and advantages of the present invention will become clearer from the following description of embodiments of the invention with reference to the accompanying drawings, in which:

[0023] Figure 1 The illustration schematically depicts application scenarios of the data import method, apparatus, device, medium, and program products according to embodiments of the present invention.

[0024] Figure 2 A flowchart illustrating a data import method for a lake according to an embodiment of the present invention is shown schematically;

[0025] Figure 3 A flowchart illustrating a data import method for a lake according to another embodiment of the present invention is shown schematically;

[0026] Figure 4 A schematic diagram illustrating the structure of a data import device according to an embodiment of the present invention is shown; and

[0027] Figure 5 A block diagram schematically illustrates an electronic device suitable for implementing the data ingestion method according to an embodiment of the present invention. Detailed Implementation

[0028] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the invention. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of the invention for ease of explanation. However, it will be apparent that one or more embodiments may be practiced without these specific details. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concept of the invention.

[0029] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the invention. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0030] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0031] When using expressions such as "at least one of A, B and C", they should generally be interpreted in accordance with the meaning that is commonly understood by those skilled in the art (e.g., "a system having at least one of A, B and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B and C, etc.).

[0032] Embodiments of the present invention provide a data ingestion method that can pre-configure the processing flow for at least three ingestion file generation methods—single-shard generation, multi-shard generation, and asynchronous generation—as well as the applicable conditions for each generation method. This allows for the selection of corresponding data tables based on the applicable conditions of each generation method, and then the corresponding ingestion files are generated through the pre-configured processing flow. In this way, data tables meeting the applicable conditions of each generation method can be processed in batches, enabling the batch and rapid generation of ingestion files, thus achieving the generalization and standardization of the ingestion file generation method and processing flow.

[0033] Figure 1 The illustration schematically depicts an application scenario of the data import method, apparatus, device, medium, and program product according to embodiments of the present invention.

[0034] like Figure 1As shown, application scenario 100 according to this embodiment may include terminal device 101, network 102, and server 103. Network 102 is used as a medium to provide a communication link between terminal device 101 and server 103. Network 102 may include various connection types, such as wired or wireless communication links or fiber optic cables, etc.

[0035] Users can use terminal device 101 to interact with server 103 via network 102 to receive or send messages, etc. Various communication client applications can be installed on terminal device 1013, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social media platform software, etc. (for example only), and various development tool software clients, etc. Terminal device 101 can be various electronic devices with a display screen and support web browsing, including but not limited to smartphones, tablets, laptops, and desktop computers, etc.

[0036] Server 103 may be a server that provides various services, such as a back-end management server that supports websites or applications accessed by users using terminal device 101 (for example only).

[0037] The data ingestion method provided in this embodiment of the invention can generally be executed by server 103. Correspondingly, the data ingestion device provided in this embodiment of the invention can generally be installed in server 103. For example, a user can interact with server 103 through terminal device 101 to configure the processing flow (such as corresponding batch jobs) for single-shard generation, multi-shard generation, and asynchronous generation methods, as well as the applicable conditions for each generation method (such as configuring an ingestion configuration file), and deploy the configuration results to server 103. Then, after the triggering conditions are met (such as timed triggering or event triggering (such as user commands)), server 103 can execute the method of this embodiment of the invention to perform data ingestion processing on tables from various data sources.

[0038] Of course, it is understood that the data ingestion method provided in the embodiments of the present invention can also be executed by a server or server cluster that is different from server 103 and is capable of communicating with terminal device 101 and / or server 103. Accordingly, the data ingestion device provided in the embodiments of the present invention can also be set in a server or server cluster that is different from server 103 and is capable of communicating with terminal device 101 and / or server 103.

[0039] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0040] It should be noted that the data import method and apparatus provided in the embodiments of the present invention can be used in the field of fintech, or in any field other than fintech. The present invention does not limit the application field.

[0041] The following will be based on Figure 1 The described scene, through Figures 2-3 The data import method for the lake according to embodiments of the present invention will be described in detail.

[0042] Figure 2 A flowchart illustrating a data import method for a lake according to an embodiment of the present invention is shown schematically.

[0043] like Figure 2 As shown, the data import method of this embodiment includes operations S210 to S230.

[0044] In operation S210, obtain the target table that meets the applicable conditions of the target generation method. The target generation method is one of the following: single-shard generation method, multi-shard generation method, or asynchronous generation method.

[0045] The single-shard generation method is applicable when the table is stored in a single-shard manner.

[0046] The asynchronous generation method is applicable when the table is stored in a multi-sharded manner, at least one first shard has more data than a threshold data volume, and the table contains a preset field. This preset field is a field that can split the data in the table into several independent parts, such as a user dimension field (e.g., user ID). For example, if the first shard contains tens of thousands of user-related records, the data in the first shard can be split into several parts based on the user ID. For instance, records for every 3000 users can be combined into one part, resulting in four parts, with the last part containing only 1000 users. The data in each split part is independent and does not affect data migration.

[0047] The applicable conditions for the multi-shard generation method include tables that are stored in a multi-sharded manner, but do not meet the applicable conditions for the asynchronous generation method.

[0048] Specifically, database sharding is a technique that breaks down a large database into multiple smaller and more manageable parts (called shards), which can be distributed across different servers or nodes to improve the system's performance, scalability, and availability.

[0049] Tables can be stored in two ways: single-shard and multi-shard.

[0050] A single shard refers to a database being divided into a single partition, with all data stored in that shard. It's suitable for scenarios with small datasets and simple queries.

[0051] Multi-sharding refers to dividing a database into multiple shards, each storing a portion of the data. Data is typically distributed across different shards based on certain rules (such as hashing or range analysis). Application scenarios include tables with large datasets and complex queries.

[0052] In this embodiment of the invention, the applicable conditions for each lake entry file generation method can be set according to the table's storage method in the database and the table's structure (such as whether there are preset fields). Specifically, tables stored in a single shard use the single shard generation method; for tables stored in multiple shards, the generation method (multi-shard generation or asynchronous generation) is selected based on whether there is a first shard with a very large amount of data and whether the table can be split into multiple independent parts using preset fields.

[0053] During operation S220, the target entry file of the target table is generated according to the processing flow of the target generation method.

[0054] The single-shard generation method processes data on a per-shard basis, directly generating lake entry files based on the data on a single shard.

[0055] The multi-shard generation method processes data on a per-shard basis, which can be divided into multiple steps. For example, a sub-inflow file can be generated for the data on each shard, and then these sub-inflow files can be merged together to form the final inflow file.

[0056] In the asynchronous generation method, data on the first shard is processed in units of data split according to a preset field. In one embodiment, when an asynchronous generation method is applied to a table, each shard in the table can be split into multiple parts according to the preset field. These multiple parts can then form a local sub-file. The local sub-files in each shard are then merged into a sub-inbound file for that shard. Finally, all the sub-inbound files from all shards are merged together to form the inbound file. In other embodiments, only data in the first shard exceeding a data volume threshold can be split into several parts according to the preset field. Each part forms a local sub-file, and these several local sub-files are then merged into a sub-file for that shard. Finally, the sub-files from multiple shards are merged together to form the inbound file. The asynchronous generation method effectively controls the amount of data processed in each generated file (whether a local sub-file or a sub-file), avoiding data congestion or excessively long generation times for a single file. Furthermore, multiple local sub-files can be generated asynchronously or in parallel, improving file generation efficiency.

[0057] In operation S230, data from the target table is imported into the data lake using the target import files. Specifically, the target import files can be loaded into the data lake to perform data extraction and import operations. In some embodiments, the target import files can be generated and then imported one by one. Alternatively, in other embodiments, the generated target import files can be temporarily stored, and after accumulating a certain number, they can be compressed into a package and sent to a specific directory. The data lake can then decompress the package from that directory and read the import files one by one to load the data.

[0058] In some embodiments of the present invention, operations S210 and S220 can be executed by a target batch job corresponding to the target generation method. Specifically, batch jobs can be configured for single-shard generation, multi-shard generation, and asynchronous generation methods, respectively, wherein the batch job configures the processing flow of the corresponding lake entry file generation method. After each batch job starts, operations S210 and S220 are executed to generate lake entry files, thus enabling batch generation of lake entry files. In some embodiments, operation S230 can also be executed by a batch job. Of course, operation S230 can also be executed without a batch job.

[0059] In one embodiment, a target entry configuration file can be pre-configured, wherein the target entry configuration file contains the table names of target tables that meet the applicable conditions of the target generation method. For example, a user can use terminal device 101 to write a target entry configuration file, configuring the table information applicable to each entry file generation method, and then upload it to server 103. In this way, during operation S210, the target batch job can quickly obtain the target table by reading the target entry configuration file, avoiding a large number of conditional judgment operations and significantly improving efficiency.

[0060] In one embodiment, a separate inbound configuration file can be configured for different inbound file generation methods. This configuration file specifically records tables that meet the applicable conditions for the corresponding inbound file generation method. In other embodiments, only a single overall inbound configuration file can be configured. This overall configuration file records each data table to be migrated and the applicable inbound file generation method for that data table. Batch jobs corresponding to single-shard generation, multi-shard generation, and asynchronous generation methods all read this overall configuration file. Each batch job can select the tables it can process from this overall configuration file based on the corresponding inbound file generation method and then perform processing.

[0061] In addition to differences in storage methods and table structures that may lead to variations in the generation of import files, the timing of data import also needs to be considered. This is because data import should minimize disruption to the production use of each table; therefore, data import should occur during periods of low or no usage for each table. Consequently, different tables can be used for data import at different times. To address this, execution time information for each table can be configured in the import configuration file. This execution time information instructs the corresponding batch jobs to generate import files only within the time frame specified for each table.

[0062] In one embodiment, batch jobs for single-shard generation, multi-shard generation, and asynchronous generation can be started at different times in different sessions. The execution time information for each table in the lake ingestion configuration file is specifically configured as session information. This way, the batch jobs in each session only process the data tables to be migrated within that session, thus achieving a universal file ingestion solution for multiple sessions and multiple generation methods. A specific embodiment can be referred to below. Figure 3 The illustration.

[0063] Figure 3 The diagram illustrates a processing flowchart of different data entry file generation methods in a data entry method according to an embodiment of the present invention.

[0064] like Figure 3 As shown, the data import method according to this embodiment may include the following steps:

[0065] S301, Initialize the lake entry configuration file. In one embodiment, the lake entry configuration file can configure information such as the number of sessions, generation method, and table name for each table. In another embodiment, a separate lake entry configuration file can be configured for different generation methods, and each lake entry configuration file specifically records information such as the table name and number of sessions for tables that meet the applicable conditions of that lake entry file generation method.

[0066] S302 configures batch jobs for each session. Each session is configured with three batch jobs based on the lake entry file generation method: single-shard generation, multi-shard generation, and asynchronous generation. Batch jobs for multiple sessions can be configured, allowing each session's batch jobs to be scheduled daily to generate lake entry files.

[0067] Next, batch jobs corresponding to different lake entry file generation methods are used to process all lake entry files in the lake entry configuration file. Specifically, the batch jobs retrieve the table names for the corresponding session and generation method from the lake entry configuration file based on the session and generation method, and generate the lake entry files for these tables. After generation, the execution results are written to the lake log.

[0068] The multi-shard generation process can include steps S311 to S313. The multi-shard generation method is suitable for tables stored in multiple shards, where the table is divided into several shards in the database, and each shard contains a portion of the data. In the multi-shard generation method, a sub-entry file for each shard is generated, and finally, the sub-entry files for each shard are merged.

[0069] Specifically, in S311, the query configuration information for the tables to be processed is obtained. For each table to be migrated, the query configuration information for that table can be pre-configured. This query configuration information includes the query statements (such as SQL statements) for that table. These query statements are used to query and extract the data that needs to be included in the lake from the table and then write it to the corresponding file.

[0070] In S312, each database shard calls the file generation component to generate sub-entry files. This file generation component is a componentized module unit that reads data based on the table's query configuration information and writes the data to the corresponding file.

[0071] In S313, the sub-entry files generated from each shard are merged into a single entry file.

[0072] The asynchronous generation process can include steps S321 through S324. Asynchronous generation is suitable for tables with multi-shard storage, where at least one shard has a very large amount of data, and there exists a field (such as a user-level field) that can divide the data into several parts. These tables are divided into several shards in the database, and each shard, or at least one shard, has a very large amount of data. The table has a field that can divide the data into several parts (e.g., if the table has a user ID field, the data can be divided into data for each user based on the user ID field; each user's data can be first merged into local sub-files, and then merged to generate the final file).

[0073] Specifically, in S321, the query configuration information for the tables to be processed is obtained. For each table to be migrated, the query configuration information for that table is pre-configured. This query configuration information includes the query statement for that table, which is used to query and extract the data that needs to be included in the lake and write it to the corresponding file.

[0074] In S322, an asynchronous mechanism is used, and each database shard generates local sub-files for each part.

[0075] In S323, multiple layout sub-files generated from each shard are merged into a single sub-lake entry file.

[0076] In S324, the sub-entry files generated from each shard are merged into a single entry file.

[0077] For example, the data in the shards is split using the user ID as a preset field. To improve generation efficiency, a local sub-file corresponding to each user is generated asynchronously based on the user ID on each shard, with each asynchronous task independent of the others. After the local sub-file for each user is generated, all user files on the shard are merged into one file. After each shard is merged, the sub-files of each shard are merged into one file to obtain the laked file.

[0078] The single-shard generation process can include steps S331 to S332. Single-shard generation is applicable to tables stored in a single shard. These tables have only one shard in the database, and this shard contains all the data. Only the entry file for this single shard needs to be generated.

[0079] In S331, the query configuration information for the tables to be processed is obtained. For each table to be migrated, the query configuration information for that table is pre-configured. This query configuration information includes the query statement for that table, which is used to query and extract the data that needs to be included in the lake and write it to the corresponding file.

[0080] In S332, the file generation component is called to generate the lake entry file.

[0081] Next, proceed with steps S304 and S305.

[0082] In S304, an alarm is triggered if any files fail to be generated. The system reads the inflow logs; if any files fail to be generated, an event alarm is sent. Alarms can be triggered via pre-configured alarm scripts.

[0083] In S306, data is imported into the data lake based on the generated import file. For example, the generated import file is sent to a specific directory, and then the file data from that directory is loaded into the data lake.

[0084] Different sessions can be configured with different data download methods, such as compressed file download. In the compressed file download method, a batch job generates the lake entry files for that session using the corresponding generation method, compresses them into a single compressed package, and then sends the compressed package to that specific directory. Other file download methods can also define the specific process for transferring files to that specific directory, which is not limited in this invention.

[0085] This invention provides an inbound configuration file that specifies the table names, inbound file generation methods, and session details for the tables to be migrated. Batch jobs can quickly and systematically inbound each table to the lake using the corresponding method by reading the inbound configuration file. If new tables are added subsequently, the inbound configuration file can be updated by adding the corresponding table information, eliminating the need to modify the batch job and effectively improving the efficiency of data inbound.

[0086] Therefore, the embodiments of the present invention can effectively improve the timeliness and versatility of data import into the lake. In some embodiments, a universal data import scheme with multiple sessions and multiple generation methods is realized through import configuration files and batch jobs. Moreover, it is easy to expand; any newly added import files can be quickly imported into the lake by updating the import configuration files, thereby improving the efficiency and scalability of data import.

[0087] Based on the data import methods of the above embodiments, this invention also provides a data import device. The following will be combined with... Figure 4 The device is described in detail.

[0088] Figure 4 A schematic block diagram of a data import device according to an embodiment of the present invention is shown.

[0089] like Figure 4 As shown, according to one embodiment of the present invention, the data import device 400 may include an acquisition module 410, a generation module 420, and an import module 430. In some embodiments, the device 400 may further include a first configuration module 440 and a second configuration module 450. The device 400 can implement a reference Figures 2-3 The data import method introduced is described.

[0090] The acquisition module 410 is used to acquire target tables that meet the applicable conditions of the target generation method. The target generation method is one of the following: single-shard generation, multi-shard generation, or asynchronous generation. The applicable conditions for the single-shard generation method include that the table is stored in a single-shard configuration. The applicable conditions for the asynchronous generation method include that the table is stored in a multi-shard configuration, and the data volume of at least one first shard is greater than a threshold data volume, and the table contains a preset field. The applicable conditions for the multi-shard generation method include that the table is stored in a multi-shard configuration, but the applicable conditions for the asynchronous generation method are not met.

[0091] The generation module 420 is used to generate the target lake entry file of the target table according to the processing flow of the target generation method. Among them, the single-shard generation method processes the data on the single shard as a unit, the multi-shard generation method processes the data on each shard as a unit, and the asynchronous generation method processes the data on the first shard as a unit of data split according to preset fields.

[0092] Import module 430 is used to import data from a target table into the data lake using the target import file.

[0093] The first configuration module 440 is configured with corresponding batch jobs for single-shard generation, multi-shard generation, and asynchronous generation methods, respectively. Correspondingly, the acquisition module 410 is specifically used to acquire target tables that meet the applicable conditions of the target generation method through the target batch job corresponding to the target generation method. The generation module 420 is specifically used to generate target lake files for the target tables by executing the processing flow of the target generation method through the target batch job corresponding to the target generation method.

[0094] The second configuration module 450 is used to configure the target entry configuration file, which contains the names of target tables that meet the applicable conditions of the target generation method. The acquisition module 410 is specifically used to read the target entry configuration file through the target batch job to obtain the target tables.

[0095] According to embodiments of the present invention, any plurality of modules among the acquisition module 410, generation module 420, import module 430, first configuration module 440, and second configuration module 450 may be combined into one module, or any one of these modules may be split into multiple modules. Alternatively, at least part of the functionality of one or more of these modules may be combined with at least part of the functionality of other modules and implemented in one module. According to embodiments of the present invention, at least one of the acquisition module 410, generation module 420, import module 430, first configuration module 440, and second configuration module 450 may be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or any other reasonable means of integrating or packaging circuitry, or implemented in hardware or firmware, or in any one of software, hardware, and firmware implementations, or in a suitable combination of any of these. Alternatively, at least one of the acquisition module 410, generation module 420, import module 430, first configuration module 440 and second configuration module 450 may be implemented at least partially as a computer program module, which can perform corresponding functions when the computer program module is run.

[0096] Figure 5 A block diagram schematically illustrates an electronic device suitable for implementing the data ingestion method according to an embodiment of the present invention.

[0097] like Figure 5As shown, an electronic device 500 according to an embodiment of the present invention includes a processor 501, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 502 or a program loaded from a storage portion 508 into a random access memory (RAM) 503. The processor 501 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 501 may also include onboard memory for caching purposes. The processor 501 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present invention.

[0098] RAM 503 stores various programs and data required for the operation of electronic device 500. Processor 501, ROM 502, and RAM 503 are interconnected via bus 504. Processor 501 executes various operations of the method flow according to embodiments of the present invention by executing programs in ROM 502 and / or RAM 503. It should be noted that the programs may also be stored in one or more memories other than ROM 502 and RAM 503. Processor 501 may also execute various operations of the method flow according to embodiments of the present invention by executing programs stored in said one or more memories.

[0099] According to an embodiment of the present invention, the electronic device 500 may further include an input / output (I / O) interface 505, which is also connected to a bus 504. The electronic device 500 may also include one or more of the following components connected to the input / output (I / O) interface 505: an input section 506 including a keyboard, mouse, etc.; an output section 507 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 508 including a hard disk, etc.; and a communication section 509 including a network interface card such as a LAN card, modem, etc. The communication section 509 performs communication processing via a network such as the Internet. A drive 510 is also connected to the input / output (I / O) interface 505 as needed. A removable medium 511, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 510 as needed so that computer programs read from it can be installed into the storage section 508 as needed.

[0100] The present invention also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs, which, when executed, implement the method according to the embodiments of the present invention.

[0101] According to embodiments of the present invention, the computer-readable storage medium may be a non-volatile computer-readable storage medium, such as including, but not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In the present invention, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to embodiments of the present invention, the computer-readable storage medium may include ROM 502 and / or RAM 503 and / or one or more memories other than ROM 502 and RAM 503 described above.

[0102] Embodiments of the present invention also include a computer program product comprising a computer program containing program code for performing the methods shown in the flowchart. When the computer program product is run on a computer system, the program code is used to cause the computer system to implement the methods provided in the embodiments of the present invention.

[0103] When the computer program is executed by the processor 501, it performs the functions defined in the system / apparatus of this invention. According to embodiments of the invention, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0104] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and may be downloaded and installed via the communication section 509, and / or installed from a removable medium 511. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0105] In such an embodiment, the computer program can be downloaded and installed from a network via communication section 509, and / or installed from removable medium 511. When the computer program is executed by processor 501, it performs the functions defined in the system of this embodiment of the invention. According to embodiments of the invention, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0106] According to embodiments of the present invention, program code for executing the computer programs provided in the embodiments of the present invention can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can be executed entirely on the user's computing device, partially on the user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0107] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0108] Those skilled in the art will understand that the features described in the various embodiments of the present invention can be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in the present invention. In particular, the features described in the various embodiments of the present invention can be combined and / or combined in various ways without departing from the spirit and teachings of the present invention. All such combinations and / or combinations fall within the scope of the present invention.

Claims

1. A method for data import into a lake, comprising: Obtain a target table that meets the applicable conditions of the target generation method; The target entry file of the target table is generated according to the processing flow of the target generation method described above; as well as The data from the target table is imported into the data lake using the target import file. The target generation method is one of the following: single-shard generation, multi-shard generation, or asynchronous generation. The applicable conditions for the single-shard generation method include that the table is stored in a single-shard manner; The applicable conditions for the asynchronous generation method include that the table is stored in a multi-sharded manner, and the data volume of at least one first shard is greater than the threshold data volume, and the table contains a preset field; The applicable conditions for the multi-shard generation method include tables that are stored in a multi-sharded manner but do not meet the applicable conditions for the asynchronous generation method; The single-shard generation method processes data on a single shard as a unit, the multi-shard generation method processes data on each shard as a unit, and the asynchronous generation method processes data on the first shard in units of data split according to the preset field.

2. The method according to claim 1, wherein, The operations of obtaining the target table and generating the target entry file are performed through a target batch job corresponding to the target generation method. Each of the single-shard generation method, multi-shard generation method, and asynchronous generation method has its own corresponding batch job configuration.

3. The method according to claim 2, wherein, The target table that satisfies the applicable conditions of the target generation method includes: The target batch job reads the target entry configuration file to obtain the target table, wherein the target entry configuration file is configured with the table name of the target table that meets the applicable conditions of the target generation method.

4. The method according to claim 3, wherein, The target lake entry configuration file also includes the execution time information for the target table, wherein... The target batch job generates the target entry file within the time range specified by the execution time information of the target table.

5. The method according to claim 4, wherein, The method further includes: At the beginning of each session, start the batch jobs corresponding to the single-shard generation method, multi-shard generation method, and asynchronous generation method in that session. The execution time information of the target table includes information on the number of sessions in which the target table is processed. The target batch job processes the target table within the number of sessions specified in the execution time information of the target table to generate the target entry file.

6. The method according to claim 3, wherein, Before obtaining the target table that meets the applicable conditions of the target generation method, the method further includes: In response to a configuration update operation that adds a new table to the target inbound configuration file, update the target inbound configuration file.

7. The method according to claim 1, wherein, The process of generating the target entry file for the target table according to the aforementioned target generation method includes: Obtain the target query configuration information for the target table; Data is read from the target table according to the query statement configured in the target query configuration information; The target lake entry file is generated based on the read data.

8. The method according to claim 1, wherein, When the target processing method is a multi-shard generation method, the step of generating the target entry file of the target table according to the processing flow of the target generation method includes: Generate a sub-inflow file for each shard of the target table; and The sub-inbound files of each segment of the target table are merged into the target inbound file.

9. The method according to claim 1, wherein, When the target processing method is asynchronous generation, generating the target lake entry file of the target table according to the processing flow of the target generation method includes: Generate a sub-inflow file for each shard of the target table; and The sub-inbound files of each segment of the target table are merged into the target inbound file; The generation of a sub-inflow file for each shard of the target table includes: The data in the first slice is split into n parts according to the preset field, where n is an integer greater than 1; Generate a local sub-file for each of the n parts; and The local sub-files corresponding to the n parts are merged to obtain the sub-entry file of the first segment.

10. A data import device, comprising: The acquisition module is used to acquire a target table that meets the applicable conditions of the target generation method; The generation module is used to generate the target entry file of the target table according to the processing flow of the target generation method; as well as The import module is used to import data from the target table into the data lake using the target import file; The target generation method is one of the following: single-shard generation, multi-shard generation, or asynchronous generation. The applicable conditions for the single-shard generation method include that the table is stored in a single-shard manner; The applicable conditions for the asynchronous generation method include that the table is stored in a multi-sharded manner, and the data volume of at least one first shard is greater than the threshold data volume, and the table contains a preset field; The applicable conditions for the multi-shard generation method include tables that are stored in a multi-sharded manner, but do not meet the applicable conditions for the asynchronous generation method. The single-shard generation method processes data on a single shard as a unit, the multi-shard generation method processes data on each shard as a unit, and the asynchronous generation method processes data on the first shard in units of data split according to the preset field.

11. An electronic device, comprising: One or more processors; Memory, used to store one or more computer programs. The one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1 to 9.

12. A computer-readable storage medium having a computer program or instructions stored thereon, wherein, When the computer program or instructions are executed by a processor, they implement the steps of the method according to any one of claims 1 to 9.

13. A computer program product comprising a computer program or instructions, wherein, When the computer program or instructions are executed by a processor, they implement the steps of the method according to any one of claims 1 to 9.

Images