Data loading method and device, computer device and computer readable storage medium

Abstract

The application relates to a data loading method and device, computer equipment, a storage medium and a computer program product. When a service system receives a data extraction request, a target SQL statement matched with the data extraction request is determined, the target SQL statement is checked, when the checking passes, the target SQL statement is executed based on a pre-set transfer database, when the target SQL statement is successfully executed, target data returned by the transfer database after executing the target SQL statement is acquired, and the target data is loaded into a target database, so that data is extracted and loaded from an EDIP. Since the embodiment extracts target data through an SQL query statement from a transfer database loaded with a database table file corresponding to data of the EDIP, the data does not need to be encoded and processed, and therefore, the data extraction and loading efficiency is greatly improved.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a data loading method, apparatus, computer equipment, storage medium, and computer program product. Background Technology

[0002] With the development of data processing technology, various data processing and data storage technologies have emerged. Enterprise Data Integration Platforms (EDIPs) have also emerged in response. Based on the information characteristics of the manufacturing industry, they provide transparent and consistent information access and interaction methods in heterogeneous distributed environments (operating systems, networks, databases), manage the applications running on them, provide services to these applications, and support the integration of application systems in various specific fields.

[0003] In traditional technologies, business systems need to obtain data from upstream systems via EDIP and load it into their own business databases. However, since the data files obtained from EDIP are usually a "mirror" of the upstream system's database table content at a certain point in time, most of the data cannot be used directly and needs to be processed through encoding.

[0004] However, with massive amounts of data, processing the data through encoding before loading it greatly affects the data loading efficiency. Summary of the Invention

[0005] Therefore, it is necessary to provide a data loading method, apparatus, computer equipment, computer-readable storage medium, and computer program product that can improve data loading efficiency in response to the above-mentioned technical problems.

[0006] Firstly, this application provides a data loading method. The method includes:

[0007] When a data extraction request is received, the target SQL statement that matches the data extraction request is determined;

[0008] The target SQL statement is validated. If the validation passes, the target SQL statement is executed based on a pre-set transit database. The transit database is obtained by loading the corresponding data from the database table file of the Enterprise Data Integration Platform (EDIP).

[0009] When the target SQL statement is executed successfully, the target data returned by the transit database after executing the target SQL statement is obtained;

[0010] The target data is loaded into the target database.

[0011] In one embodiment, the method further includes: obtaining a data file corresponding to the table name from EDIP and storing it in a local cache directory based on a pre-configured table name and file cache directory for the data to be loaded; reading each header file from the local cache directory, identifying the set of header field information in the header file, and obtaining an original hash table with the table name as the key and the set of header field information as the value; filtering the original hash table according to the table name of the data to be loaded and the required column names to obtain a target hash table; and loading the corresponding data into the transit database according to the target hash table.

[0012] In one embodiment, loading corresponding data into the transit database according to the target hash table includes: determining whether a corresponding table name exists in the transit database based on the table name of each table in the target hash table; if a corresponding table name exists in the transit database, determining whether the fields in the table corresponding to the table name can cover the set of field information corresponding to the table name in the target hash table; if they cannot cover each other, filtering out an uncovered subset of target fields from the target hash table, and updating the table structure corresponding to the table name in the transit database according to the subset of target fields.

[0013] In one embodiment, the method further includes: generating a log record of the target SQL statement when the target SQL statement fails to execute.

[0014] In one embodiment, the verification of the target SQL statement includes: parsing the operation mode of the target SQL statement; when the operation mode is a query operation or a calculation operation, the verification passes; when the operation mode is a data addition operation or a data modification operation, determining the operation range of the target SQL statement, and when the operation range matches the target operation range, the verification passes.

[0015] In one embodiment, the data extraction request carries a data extraction identifier and a parameter group; determining the target SQL statement that matches the data extraction request includes: determining a matching target SQL statement template from a pre-configured SQL statement template based on the data extraction identifier; filling the parameter group in the corresponding position according to the parameter placeholder information pre-set in the target SQL statement template, and generating a target SQL statement that matches the data extraction request.

[0016] Secondly, this application also provides a data loading apparatus. The apparatus includes:

[0017] The SQL statement determination module is used to determine the target SQL statement that matches the data extraction request when a data extraction request is received.

[0018] The SQL statement verification module is used to verify the target SQL statement. When the verification passes, the target SQL statement is executed based on a pre-set transit database. The transit database is obtained by loading corresponding data from the database table file of the Enterprise Data Integration Platform (EDIP).

[0019] The data acquisition module is used to acquire the target data returned by the transit database after the target SQL statement is executed when the target SQL statement is executed successfully;

[0020] The data loading module is used to load the target data into the target database.

[0021] Thirdly, this application also provides a computer device. The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the steps of the method described in the first aspect above.

[0022] Fourthly, this application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, implements the steps of the method described in the first aspect above.

[0023] Fifthly, this application also provides a computer program product. The computer program product includes a computer program that, when executed by a processor, implements the steps of the method described in the first aspect above.

[0024] The aforementioned data loading method, apparatus, computer equipment, storage medium, and computer program product, when the business system receives a data extraction request, determine the target SQL statement matching the data extraction request and validate the target SQL statement. If the validation passes, the target SQL statement is executed based on a pre-set transit database. When the target SQL statement is successfully executed, the target data returned by the transit database after executing the target SQL statement is obtained, thereby loading the target data into the target database. This achieves data extraction and loading from EDIP. Since this embodiment extracts the target data from a transit database containing the corresponding data in the database table file loaded with EDIP through an SQL query statement, no further data encoding or processing is required, thus greatly improving the efficiency of data extraction and loading. Attached Figure Description

[0025] Figure 1 This is a flowchart illustrating a data loading method in one embodiment;

[0026] Figure 2 This is a flowchart illustrating the steps for generating a transit database in another embodiment;

[0027] Figure 3 This is a flowchart illustrating the steps of transferring data to a relay data device in one embodiment;

[0028] Figure 4 This is a structural block diagram of a data loading device in one embodiment;

[0029] Figure 5 This is an application environment diagram of the data loading method in one embodiment;

[0030] Figure 6 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0031] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0032] In one embodiment, such as Figure 1 As shown, a data loading method is provided. Taking a business system that needs to load data from EDIP as an example, the method is explained in detail, including the following steps:

[0033] Step 102: When a data extraction request is received, determine the target SQL statement that matches the data extraction request.

[0034] The data extraction request is a request received by the business system to extract data from the EDIP. The target SQL (Structured Query Language) statement is the command that matches the data extraction request to operate on the database to query the corresponding data.

[0035] In this embodiment, when the business system receives a data extraction request, it can parse the data extraction request to determine the target SQL statement that matches the data extraction request.

[0036] Specifically, the data extraction request carries a data extraction identifier and a parameter set. Determining the target SQL statement that matches the data extraction request can include: identifying the matching target SQL statement template from a pre-configured SQL statement template based on the data extraction identifier; and filling in the parameter set in the corresponding positions according to the pre-set parameter placeholder information in the target SQL statement template to generate the target SQL statement that matches the data extraction request.

[0037] The pre-configured SQL statement templates are pre-configured SQL query statements corresponding to different data extraction ranges. The target SQL statement template is the SQL query statement corresponding to the data extraction request. The data extraction identifier is a unique marker or label used to determine the data extraction range; therefore, each data extraction identifier has a corresponding SQL statement template, meaning each data extraction identifier represents a corresponding data extraction range. Specifically, the SQL statement template also allows users to fill in parameter placeholder information; that is, parameter placeholder information is pre-set in the SQL statement template. Therefore, in this embodiment, the business system can determine the matching target SQL statement template from the pre-configured SQL statement templates based on the data extraction identifier carried in the received data extraction request, and fill in the elements of the parameter group into the corresponding positions according to the pre-set parameter placeholder information in the target SQL statement template, thereby generating the target SQL statement matching the data extraction request.

[0038] Step 104: Validate the target SQL statement. If the validation passes, execute the target SQL statement based on the pre-configured transit database.

[0039] The transit database is obtained by loading corresponding data from the database table files of the Enterprise Data Integration Platform (EDIP). Specifically, the verification can check the legality of the operation scope and operation method of the target SQL statement. Verification passes when both the operation scope and operation method are legal, and fails when either is invalid. In this embodiment, the business system can also verify the aforementioned target SQL statement. If the verification passes, the target SQL statement is executed based on the pre-set transit database; that is, the target SQL statement is run in the pre-set transit database.

[0040] Specifically, in one scenario, the target SQL statement is validated. This validation may include: parsing the operation method of the target SQL statement; if the operation method is a query operation or a calculation operation, the validation passes; if the operation method is a data insertion operation or a data modification operation, the operation scope of the target SQL statement is further determined. If the operation scope matches the target operation scope, the validation passes. If the operation scope does not match the target operation scope, the validation fails. The target operation scope can be the range of table objects that the target SQL statement is allowed to execute, for example, it could be a temporary table.

[0041] Step 106: When the target SQL statement is executed successfully, obtain the target data returned by the transit database after executing the target SQL statement.

[0042] The target data is the data returned by the transit database after successfully executing the target SQL statement, and it is also the data that needs to be extracted corresponding to the data extraction request.

[0043] In this embodiment, when the target SQL statement is successfully executed by the relay database, the business system can also obtain the target data returned by the relay database after executing the target SQL statement, that is, obtain the data that needs to be extracted.

[0044] Step 108: Load the target data into the target database.

[0045] The target database can be a database integrated into the business system, or it can be a cloud data platform that provides database services to the business system. Specifically, the business system can load the target data obtained in the above steps into the target database, thereby realizing the extraction and loading of data from EDIP.

[0046] In the above data loading method, when the business system receives a data extraction request, it determines the target SQL statement that matches the request and validates it. If the validation passes, the target SQL statement is executed based on a pre-set transit database. When the target SQL statement is successfully executed, the target data returned by the transit database is obtained, and thus the target data is loaded into the target database. This achieves data extraction and loading from EDIP. Since this embodiment extracts the target data from a transit database containing the corresponding data in the database table file of the EDIP through an SQL query, no further data encoding or processing is required, thus greatly improving the efficiency of data extraction and loading. Furthermore, through the SQL query, data can be intelligently retrieved based on the fields that actually need to be loaded, thereby reducing the entry of redundant data and improving loading performance.

[0047] In one embodiment, the method may further include: generating a log record of the target SQL statement when the target SQL statement fails to execute. Specifically, when the target SQL statement fails to execute due to being relayed to an intermediate database, the business system may also generate a log record of the target SQL statement, that is, record the failed SQL statement in a log file. This facilitates the correction and re-execution of the failed statement and improves the success rate of subsequent SQL statement execution.

[0048] In one embodiment, such as Figure 2 As shown, the above method may further include:

[0049] Step 202: Based on the pre-configured table name and local cache directory for the data to be loaded, periodically retrieve the data file corresponding to the table name from EDIP and store it in the local cache directory.

[0050] The data files include header files and data files. In this embodiment, the business system can pre-configure the table names and corresponding local cache directories for the header and data files to be loaded, and then download the latest data files (including header and data files) corresponding to the table names from EDIP to the local cache directory.

[0051] In one scenario, when reading the data file corresponding to the table name from EDIP into the local cache, the maximum number of rows that can be read at once can be pre-configured. For example, if the pre-configured maximum number of rows to read at once is N, then at most N rows of data will be read from the table data file into memory each time. After the corresponding data is loaded into the intermediate database according to subsequent steps, the next data read will be performed. This avoids the performance impact on the business system when reading large amounts of data files.

[0052] Furthermore, the header files and data files involved in the data reading operation can be marked as read, thereby avoiding repeated readings.

[0053] Step 204: Read each header file from the local cache directory, identify the set of header field information in the header file, and obtain the original hash table with the table name as the key and the set of header field information as the value.

[0054] Specifically, the business system can read the various table header files in the local cache directory. Each header file contains the table name corresponding to that header and information describing the various fields of the data table. The field information includes the field name, field length, and field type. This information is read out as a set of header field information, with each element of the set representing a field. Then, a raw hash table is generated with the table name as the key and the set of header field information as the value.

[0055] Step 206: Filter the original hash table according to the table name of the data to be loaded and the required column names to obtain the target hash table.

[0056] The target hash table is the hash table corresponding to the data that ultimately needs to be loaded. Specifically, the business system can also filter the original hash table based on the pre-configured table names to be loaded and the required column names to obtain the filtered target hash table.

[0057] Step 208: Load the corresponding data into the transit database according to the target hash table.

[0058] In this embodiment, the business system loads the corresponding data into the relay database based on the target hash table, thereby loading the data from the EDIP into the relay database. When the business system needs to use the data, it can extract the data from the relay database by executing SQL statements, thus avoiding the performance impact on the business system when loading large amounts of data. The business system actively issuing SQL processing statements to the relay database to obtain the resulting business data also avoids the security impact of SQL injection on the business system.

[0059] In one scenario, Figure 2 The process shown can precede Figure 1 The process follows a set procedure, but it is not mandatory, and this embodiment does not limit it.

[0060] Specifically, when loading the corresponding data into the relay database, a list of data insertion SQL statements can be constructed based on the field information and values ​​of each element in the target hash table. This list is then sent to the relay database to insert the data. If the relay database returns an error, the executed data insertion SQL statement is logged, and the process continues. If no error occurs, the process continues.

[0061] In one embodiment, such as Figure 3 As shown, in step 212, the corresponding data is loaded into the transit database according to the target hash table, which may further include:

[0062] Step 302: Determine whether the corresponding table name exists in the transit database based on the table name of each table in the target hash table.

[0063] Specifically, based on the table name of each table in the target hash table, query the transit database to see if there is a table with the same name. For example, the query can be performed by constructing an SQL statement to check the table structure. If the query determines that there is no table with the same name in the transit database, then step 304 is executed. If the query determines that there is a table with the same name in the transit database, then step 306 is executed.

[0064] Step 304: Create the corresponding table in the transit database based on the table name.

[0065] Specifically, if the corresponding table name does not exist in the transit database, the corresponding table will be created in the transit database based on the table name. For example, this can be done by constructing a table creation SQL statement and then creating the corresponding table in the transit database based on that SQL statement.

[0066] In one scenario, in order to store data with multiple time dimensions in a single table, the constructed table creation SQL statement can also include a column representing the data export time, thereby marking the data with multiple time dimensions.

[0067] Step 306: Determine whether the fields in the table corresponding to the table name can cover the set of field information corresponding to the table name in the target hash table.

[0068] Specifically, if a corresponding table name exists in the transit database, it is further determined whether the fields in the table corresponding to the table name can cover the set of field information corresponding to the table name in the target hash table. If they cannot cover each other, step 308 is executed; if they can cover each other, step 310 is executed.

[0069] Step 308: Filter out the subset of target fields that have not been covered from the target hash table, and update the table structure corresponding to the table name in the transit database based on the subset of target fields.

[0070] Specifically, if overwriting is not possible, a subset of target fields that are not covered is selected from the target hash table. This means selecting a subset of target fields from the set of field information corresponding to the table name in the target hash table that are missing fields compared to the corresponding fields in the intermediate database. The table structure corresponding to the table name in the intermediate database is then updated based on this subset of target fields. For example, an SQL statement to add a table field can be constructed, and this statement can be used to add the corresponding subset of target fields to the corresponding table in the intermediate database, thereby updating the table structure corresponding to the table name in the intermediate database.

[0071] Step 310: End the process.

[0072] Specifically, if the coverage is complete, the process ends.

[0073] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0074] Based on the same inventive concept, this application also provides a data loading apparatus for implementing the data loading method described above. The solution provided by this apparatus is similar to the implementation described in the above method; therefore, the specific limitations in one or more data loading apparatus embodiments provided below can be found in the limitations of the data loading method described above, and will not be repeated here.

[0075] In one embodiment, such as Figure 4 As shown, a data loading device is provided, including: an SQL statement determination module 402, an SQL statement verification module 404, a data acquisition module 406, and a data loading module 408, wherein:

[0076] SQL statement determination module 402 is used to determine the target SQL statement that matches the data extraction request when a data extraction request is received;

[0077] The SQL statement verification module 404 is used to verify the target SQL statement. When the verification passes, the target SQL statement is executed based on a pre-set transit database. The transit database is obtained by loading corresponding data from the database table file of the Enterprise Data Integration Platform (EDIP).

[0078] The data acquisition module 406 is used to acquire the target data returned by the transit database after executing the target SQL statement when the target SQL statement is executed successfully.

[0079] The data loading module 408 is used to load the target data into the target database.

[0080] In one embodiment, the apparatus further includes a transit database generation module, configured to: periodically retrieve data files corresponding to the table names from the Enterprise Data Integration Platform (EDIP) and store them in the local cache directory based on pre-configured table names of the data to be loaded and the local cache directory; wherein the data files include header files and table data files; the business system can read each header file in the local cache directory, which contains the table name corresponding to the header and information describing each field of the data table, including field name, field length, and field type. This information is read to form a header field set, where each element of the set represents a field. Next, a raw hash table is generated with the table name as the key and the header field information set as the value; the raw hash table is filtered according to the table name of the data to be loaded and the required column names to obtain a target hash table; and the corresponding data is loaded according to the target hash table to generate the transit database.

[0081] In one embodiment, the transit database generation module is further configured to: determine whether a corresponding table name exists in the transit database based on the table name of each table in the target hash list; if no corresponding table name exists in the transit database, create a corresponding table in the transit database based on the table name; if a corresponding table name exists in the transit database, determine whether the fields in the table corresponding to the table name can cover the set of field information corresponding to the table name in the target hash list; if they cannot cover, filter out an uncovered subset of target fields from the target hash list, and update the table structure in the transit database corresponding to the table name based on the subset of target fields.

[0082] In one embodiment, the apparatus further includes a logging module for generating a log record of the target SQL statement when the target SQL statement fails to execute.

[0083] In one embodiment, the SQL statement verification module is used to: parse the operation mode of the target SQL statement; when the operation mode is a query operation or a calculation operation, the verification passes; when the operation mode is a data addition operation or a data modification operation, determine the operation range of the target SQL statement; when the operation range matches the target operation range, the verification passes.

[0084] In one embodiment, the data extraction request carries a data extraction identifier and a parameter group; the SQL statement determination module is used to: determine a matching target SQL statement template from a pre-configured SQL statement template according to the data extraction identifier; fill the parameter group in the corresponding position according to the parameter placeholder information pre-set in the target SQL statement template, and generate a target SQL statement that matches the data extraction request.

[0085] In one embodiment, such as Figure 5 As shown, the data loading principle of this application is further explained in conjunction with the above-described data loading method and data loading device. Specifically, it can be applied to, for example... Figure 5 The application environment shown. Among them,

[0086] An Enterprise Data Platform (EDIP, or Enterprise Data Integration Platform) is based on enterprise information characteristics and provides transparent and consistent information access and interaction methods for enterprises in a heterogeneous distributed environment. It manages the applications running on the platform, provides services to these applications, and supports the integration of application systems in specific domains. Business systems, on the other hand, define the business processes required for the enterprise to achieve its positioning, the roles played by various partners, and the methods and content of cooperation and transactions among stakeholders.

[0087] Business systems often need to retrieve data from upstream systems via EDIP and load it into their own business databases. However, in traditional technologies, the data files retrieved from EDIP are usually a "mirror" of the upstream system's database table content at a certain point in time. This means that most of the data cannot be used directly and needs to be processed through encoding, resulting in low data loading efficiency.

[0088] Based on this, this application establishes a relay database module (i.e., a relay database) between the business system and the EDIP. Data files from the EDIP are first loaded into the relay database. When the business system needs to use the data, it extracts the data from the relay database by executing SQL statements. Both the data loading program and the relay database can be deployed on computers outside the business system, thus avoiding the performance impact on the business system when loading large amounts of data. The business system actively sends SQL processing statements to the relay database to obtain the resulting business data, also preventing SQL injection attacks from affecting the security of the business system.

[0089] Specifically, such as Figure 5 As shown, the process of loading data information files from EDIP into the transit database is jointly completed by the table file parsing module, the table data loading module, and the transit database module. The process of data extraction initiated by the business system is jointly completed by the data extraction module, the script execution module, and the transit database module.

[0090] The system comprises several modules: The table file parsing module receives the database header file (flg) and database table data file (dat) from the upstream system of the Enterprise Data Platform (EDIP), parses them, and stores them in a hash table. The table data loading module constructs SQL (Database Manipulation Language) statements from the data in the hash table and sends them to the intermediate database module for data insertion. The intermediate database module, composed of MySQL (a relational database management system), stores the data. The data extraction module receives data extraction requests from business systems, matches the fields in the requests with the corresponding SQL statements for data extraction based on user configuration, and then passes them to the script execution module. The script execution module receives the SQL statements from the data extraction module, sends them to the intermediate database module to obtain the results, and returns them.

[0091] The table file parsing module, table data loading module, and intermediate database module can be deployed and run as separate programs, independent of the business system, to isolate the impact of loading operations on the business system. The data extraction module and script execution module, on the other hand, can be integrated with the business system as a plugin SDK (Software Development Kit), providing services to the business system through interfaces.

[0092] Each module in the aforementioned data loading device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of a computer device in hardware form or independent of it, or stored in the memory of the computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0093] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 6 As shown, this computer device includes a processor, memory, input / output interfaces (I / O), and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores data retrieved from EDIP and SQL statement data. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network connection. When executed by the processor, the computer program implements a data loading method.

[0094] Those skilled in the art will understand that Figure 6 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0095] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the method described above.

[0096] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described above.

[0097] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps of the method described above.

[0098] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0099] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0100] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0101] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A data loading method, characterized in that, The method includes: When a data extraction request is received, the target SQL statement that matches the data extraction request is determined; The target SQL statement is validated. If the validation passes, the target SQL statement is executed based on a pre-set transit database. The transit database is obtained by loading the corresponding data from the database table file of the Enterprise Data Integration Platform (EDIP). When the target SQL statement is executed successfully, the target data returned by the transit database after executing the target SQL statement is obtained; Load the target data into the target database; The method further includes: Based on the pre-configured table name and file cache directory of the data to be loaded, retrieve the data file corresponding to the table name from EDIP and store it in the local cache directory; Read each header file from the local cache directory, identify the set of header field information in the header file, and obtain an original hash table with the table name as the key and the set of header field information as the value; Based on the table name and required column names of the data to be loaded, the original hash table is filtered to obtain the target hash table; Loading corresponding data into the transit database according to the target hash table includes: determining whether a corresponding table name exists in the transit database based on the table name of each table in the target hash table; if the corresponding table name does not exist in the transit database, creating a corresponding table in the transit database based on the table name; if the corresponding table name exists in the transit database, determining whether the fields in the table corresponding to the table name can cover the set of field information corresponding to the table name in the target hash table; if they cannot cover each other, filtering out an uncovered subset of target fields from the target hash table, and updating the table structure in the transit database corresponding to the table name based on the subset of target fields.

2. The method according to claim 1, characterized in that, The method further includes: When the target SQL statement fails to execute, a log record of the target SQL statement is generated.

3. The method according to any one of claims 1 to 2, characterized in that, The validation of the target SQL statement includes: The operation method for parsing the target SQL statement; The verification passes when the operation is a query or calculation operation. When the operation mode is a data addition operation or a data modification operation, the operation range of the target SQL statement is determined. When the operation range matches the target operation range, the verification passes.

4. The method according to any one of claims 1 to 2, characterized in that, The data extraction request carries a data extraction identifier and a parameter group; determining the target SQL statement that matches the data extraction request includes: Based on the data extraction identifier, determine the matching target SQL statement template from the pre-configured SQL statement templates; Based on the pre-set parameter placeholder information in the target SQL statement template, the parameter group is filled in the corresponding position to generate a target SQL statement that matches the data extraction request.

5. A data loading device, characterized in that, The device includes: The SQL statement determination module is used to determine the target SQL statement that matches the data extraction request when a data extraction request is received. The SQL statement verification module is used to verify the target SQL statement. When the verification passes, the target SQL statement is executed based on a pre-set transit database. The transit database is obtained by loading corresponding data from the database table file of the Enterprise Data Integration Platform (EDIP). The data acquisition module is used to acquire the target data returned by the transit database after the target SQL statement is executed when the target SQL statement is executed successfully; The data loading module is used to load the target data into the target database; The device further includes a transit database generation module, configured to: obtain data files corresponding to the table names from EDIP and store them in a local cache directory based on pre-configured table names and file cache directories of the data to be loaded; read each table header file from the local cache directory, identify the set of table header field information in the table header files, and obtain an original hash table with the table names as keys and the set of table header field information as values; filter the original hash table based on the table names of the data to be loaded and the required column names to obtain a target hash table; and load the corresponding data into the transit database based on the target hash table, including: determining whether a corresponding table name exists in the transit database based on the table names of each table in the target hash table; if no corresponding table name exists in the transit database, creating a corresponding table in the transit database based on the table name; if a corresponding table name exists in the transit database, determining whether the fields in the table corresponding to the table name can cover the set of field information corresponding to the table name in the target hash table; if they cannot cover each other, filtering out an uncovered target field subset from the target hash table, and updating the table structure corresponding to the table name in the transit database based on the target field subset.

6. The apparatus according to claim 5, characterized in that, The device further includes a log recording module, used to generate a log record of the target SQL statement when the target SQL statement fails to execute.

7. The apparatus according to any one of claims 5 to 6, characterized in that, The SQL statement verification module is used to: parse the operation mode of the target SQL statement; when the operation mode is a query operation or a calculation operation, the verification passes; when the operation mode is a data addition operation or a data modification operation, determine the operation range of the target SQL statement; when the operation range matches the target operation range, the verification passes.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 4.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 4.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 4.

Images