Dynamic unloading method and device for data

Abstract

The invention relates to the technical field of data processing, and discloses a dynamic unloading method and device for data. The method comprises the following steps: acquiring a mapping configuration file in a database; performing data verification on the mapping configuration file; after the mapping configuration file passes the data verification, creating an unloading appearance according to a field name and a field type in the mapping configuration file; judging whether a complex unloading scene exists at present or not according to association table information in the mapping configuration file; if the complex unloading scene exists at present, creating a temporary table according to the association table information, and inserting an unloading appearance from the temporary table to perform data unloading; and if the complex unloading scene does not exist at present, inserting an unloading appearance from the physical table where the to-be-unloaded data is located to perform data unloading. According to the method, flexible data unloading is realized, the scene of complex data unloading is met, the defect that a traditional appearance can only be exported in a single form is overcome, and the use mode of the appearance is widened.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular to a method and device for dynamically unloading data. Background Art

[0002] At present, external tables are often used to unload data and export data from the database to data files. In the process of parallel data export, it is necessary to create external tables to help the data service tool GDS to specify the exported data files. External tables store information such as the location of the data file, file format, storage location, encoding format, and separators between data.

[0003] However, the current data unloading method has the following disadvantages: 1. Difficult to expand: The external table format, field order, file type, and storage location created by the current unloading method are relatively fixed and cannot be adjusted according to the personalized field requirements of the downstream system; 2. Data security is difficult to guarantee: Some downstream systems do not need to use customer sensitive information, and the current data unloading method cannot desensitize sensitive fields, resulting in unnecessary fields being pushed to the downstream system; 3. Waste of storage resources: Some downstream systems do not need to access Chinese fields, and the current data unloading method cannot filter the fields, resulting in unnecessary waste of disk space in the downstream system; 4. Long development cycle and poor timeliness response: Some data interface requirements are customized data summaries of existing data warehouse models. The current data unloading method can only download detailed data or develop new data warehouse models, resulting in the overall data development, testing, joint debugging, and production cycle being extended; 5. Lack of flexibility: The current data unloading method only supports single-table export, does not support multi-table related export, cannot meet temporary complex unloading scenarios, and increases the complexity of downstream system development. Summary of the invention

[0004] The present invention provides a method and device for dynamically unloading data, which realizes flexible data unloading, satisfies the scenario of complex data unloading, solves the defect that traditional tables can only export single tables, and broadens the usage of tables.

[0005] In order to solve the above technical problems, the present invention provides a method for dynamically unloading data, comprising:

[0006] Obtaining a mapping configuration file in a database; wherein the mapping configuration file is generated according to the uninstallation requirements of the data to be uninstalled;

[0007] Performing data verification on the mapping configuration file; wherein the data verification includes primary key uniqueness verification, key field non-empty verification and processing rule standardization verification;

[0008] When the mapping configuration file passes the data verification, an uninstallation table is created according to the field name and field type in the mapping configuration file;

[0009] Determine whether there is a complex uninstallation scenario currently according to the association table information in the mapping configuration file;

[0010] If there is a complex unloading scenario, a temporary table is created according to the associated table information, and the unloading outer table is inserted from the temporary table to perform data unloading;

[0011] If there is no complex unloading scenario currently, the unloading outer table is inserted from the physical table where the data to be unloaded is located to perform data unloading.

[0012] Furthermore, the mapping configuration file is specifically:

[0013] According to the unloading requirements of the data to be unloaded, determine the content corresponding to each column name in the mapping configuration file and generate the mapping configuration file;

[0014] The names of the columns in the mapping configuration file include object name, field name, field description, field type, mapping rule, mode name, association table, association type, association condition, filter condition and association table alias.

[0015] Furthermore, the data verification of the mapping configuration file is specifically performed as follows:

[0016] Performing primary key uniqueness check on each column name in the mapping configuration file, and when there are duplicate column names, determining that the mapping configuration file data check fails;

[0017] Perform non-empty check on key fields of the mapping configuration file. When there are empty values ​​in the field name, field type, mapping rule, association table, association type, association condition, filter condition and association table alias in the mapping configuration file, it is determined that the data check of the mapping configuration file fails.

[0018] The mapping configuration file is checked for processing rule compliance. When a field is missing or there are two or more filter conditions, it is determined that the mapping configuration file data check fails.

[0019] Furthermore, when the mapping configuration file passes the data verification, an uninstallation table is created according to the field name and field type in the mapping configuration file, specifically:

[0020] Get the field name and field type in the mapping configuration file;

[0021] Generate a table creation statement according to the field name and the field type;

[0022] Create an uninstallation table according to the table creation statement.

[0023] Further, judging whether there is a complex uninstallation scenario currently according to the association table information in the mapping configuration file is specifically:

[0024] Get the number of association tables and the file contents of each association table according to the association table information in the mapping configuration file;

[0025] When the number of the association tables is two or more, or the file content contains a subquery, it is determined that a complex uninstallation scenario currently exists;

[0026] When the number of the association table is one and the file content does not contain a subquery, it is determined that no complex uninstallation scenario currently exists.

[0027] Furthermore, if there is a complex uninstallation scenario, a temporary table is created according to the association table information, specifically:

[0028] A temporary table is created, and the file contents of each of the associated tables are stored in the temporary table.

[0029] Furthermore, before inserting the unloading table to unload data, the method further includes:

[0030] Performing GDS detection on the uninstallation table, and when the uninstallation table passes the GDS detection, using the uninstallation table to perform data uninstallation;

[0031] Wherein, the GDS detection is specifically:

[0032] Acquire a date parameter according to an uninstallation requirement of the data to be uninstalled; wherein the date parameter corresponds to the data to be uninstalled;

[0033] Get the process information corresponding to the GDS service background on the local server;

[0034] Obtain date information from the process information to obtain a GDS specified date;

[0035] Comparing the date parameter with the GDS specified date;

[0036] When the date parameter is consistent with the GDS specified date, it is determined that the uninstallation form has passed the inspection.

[0037] Furthermore, the mapping configuration file is obtained from the database, specifically:

[0038] Automatically connect to the database and obtain the mapping configuration file in the relational database.

[0039] Furthermore, the method for dynamically unloading data further includes:

[0040] After the data is uninstalled, the uninstallation table is deleted.

[0041] The present invention provides a method for dynamic data unloading, which includes obtaining a mapping configuration file in a database; performing data verification on the mapping configuration file; creating an unloading table according to the field name and field type in the mapping configuration file after the mapping configuration file passes the data verification; judging whether a complex unloading scenario currently exists according to the associated table information in the mapping configuration file; if a complex unloading scenario currently exists, creating a temporary table according to the associated table information, and inserting the unloading table from the temporary table to unload the data; if no complex unloading scenario currently exists, inserting the unloading table from the physical table where the data to be unloaded is located to unload the data. The present invention realizes flexible data unloading, meets the scenario of complex data unloading, solves the defect that the traditional table can only export a single table, broadens the use of the table, reduces the complexity of data development of the data warehouse and downstream system, and reduces the overall development cycle.

[0042] Accordingly, the present invention provides a data dynamic unloading device, comprising: an acquisition module, a verification module, an appearance creation module, a judgment module, a first unloading module and a second unloading module;

[0043] The acquisition module is used to acquire a mapping configuration file in a database; wherein the mapping configuration file is generated according to the uninstallation requirements of the data to be uninstalled;

[0044] The verification module is used to perform data verification on the mapping configuration file; wherein the data verification includes primary key uniqueness verification, key field non-empty verification and processing rule standardization verification;

[0045] The appearance creation module is used to create an uninstall appearance according to the field name and field type in the mapping configuration file after the mapping configuration file passes the data verification;

[0046] The judging module is used to judge whether there is a complex uninstallation scenario currently according to the association table information in the mapping configuration file;

[0047] The first unloading module is used to create a temporary table according to the association table information if there is a complex unloading scenario, and insert the unloading outer table from the temporary table to perform data unloading;

[0048] The second unloading module is used for inserting the unloading outer table into the physical table where the data to be unloaded is located to unload the data if there is no complex unloading scenario currently.

[0049] The present invention provides a dynamic data unloading device, which is based on the organic combination between modules, obtains a mapping configuration file in a database; performs data verification on the mapping configuration file; after the mapping configuration file passes the data verification, creates an unloading table according to the field name and field type in the mapping configuration file; determines whether a complex unloading scenario currently exists according to the associated table information in the mapping configuration file; if a complex unloading scenario currently exists, creates a temporary table according to the associated table information, and inserts the unloading table from the temporary table to unload the data; if there is no complex unloading scenario currently, inserts the unloading table from the physical table where the data to be unloaded is located to unload the data. The present invention realizes flexible data unloading, meets the scenario of complex data unloading, solves the defect that traditional tables can only export single tables, broadens the use of tables, reduces the complexity of data development of data warehouses and downstream systems, and reduces the overall development cycle. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 A schematic diagram of a flow chart of an embodiment of a method for dynamically unloading data provided by the present invention;

[0051] Figure 2 A schematic diagram of the system architecture of an embodiment of the dynamic data unloading system provided by the present invention;

[0052] Figure 3 A schematic diagram of a flow chart of another embodiment of the method for dynamically unloading data provided by the present invention;

[0053] Figure 4 A schematic structural diagram of an embodiment of a device for dynamically unloading data provided by the present invention. DETAILED DESCRIPTION

[0054] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0055] The flowcharts shown in the accompanying drawings are only examples and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may also be decomposed, combined or partially merged, so the actual execution order may change according to actual conditions.

[0056] Some embodiments of the present invention are described in detail below in conjunction with the accompanying drawings. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0057] Example 1

[0058] See also Figure 1 , is a flow chart of an embodiment of a method for dynamic data unloading provided by the present invention, the method includes steps 101 to 106, each step is specifically as follows:

[0059] Step 101: Obtain a mapping configuration file in a database; wherein the mapping configuration file is generated according to the uninstallation requirements of the data to be uninstalled.

[0060] Furthermore, in the first embodiment of the present invention, the mapping configuration file is specifically:

[0061] According to the unloading requirements of the data to be unloaded, determine the content corresponding to each column name in the mapping configuration file and generate the mapping configuration file;

[0062] The names of the columns in the mapping configuration file include object name, field name, field description, field type, mapping rule, mode name, association table, association type, association condition, filter condition and association table alias.

[0063] Further, in the first embodiment of the present invention, the mapping configuration file is obtained in the database, specifically:

[0064] Automatically connect to the database and obtain the mapping configuration file in the relational database.

[0065] In the first embodiment of the present invention, a mapping configuration file is stored in a relational database. The mapping configuration file can unload any form of data to be unloaded. By filling in data unloading requirements such as direct unloading of a single table and unloading after logical processing into the mapping configuration file, any form of data unloading can be achieved. When using the mapping configuration file to unload the data to be unloaded, the database is automatically connected first to obtain the mapping configuration file information. Among them, the mapping configuration file includes multiple column names, including object name, field name, field description, field type, mapping rule, mode name, association table, association type, association condition, filter condition and association table alias. The present invention uses the mapping configuration file to unload data, which is not limited to the data parameters of the data to be unloaded stored in the template database. It can also add a logical processing link according to the data parameters before the data is unloaded, so that the data unloading method is more flexible.

[0066] Step 102: Perform data verification on the mapping configuration file; wherein the data verification includes primary key uniqueness verification, key field non-empty verification and processing rule standardization verification.

[0067] Furthermore, in the first embodiment of the present invention, data verification is performed on the mapping configuration file, specifically:

[0068] Performing primary key uniqueness check on each column name in the mapping configuration file, and when there are duplicate column names, determining that the mapping configuration file data check fails;

[0069] Perform non-empty check on key fields of the mapping configuration file. When there are empty values ​​in the field name, field type, mapping rule, association table, association type, association condition, filter condition and association table alias in the mapping configuration file, it is determined that the data check of the mapping configuration file fails.

[0070] The mapping configuration file is checked for processing rule compliance. When a field is missing or there are two or more filter conditions, it is determined that the mapping configuration file data check fails.

[0071] In the first embodiment of the present invention, after obtaining the mapping configuration file, before unloading the data, the mapping configuration file is first verified to ensure the integrity of the data and improve the security of data unloading. Among them, the data verification includes primary key uniqueness verification, key field non-empty verification and processing rule standardization verification. The primary key uniqueness verification verifies the uniqueness of the field name of the mapping configuration file and verifies whether there are duplicate values. The key field non-empty verification verifies the field name, field type, mapping rule, association table, association type, association condition, filter condition and association table alias to ensure that the above information is not an empty value. The processing rule standardization verification includes two aspects of verification, one of which is that the field mapping information should be all empty or all not empty; the other is that there cannot be multiple filter conditions. The present invention can avoid unloading anomalies in the unloading process in advance by performing data verification on the mapping configuration file, such as abnormal situations such as missing tables and missing fields in logical processing.

[0072] Step 103: When the mapping configuration file passes the data verification, an uninstallation table is created according to the field name and field type in the mapping configuration file.

[0073] Further, in the first embodiment of the present invention, after the mapping configuration file passes the data verification, an uninstallation table is created according to the field name and field type in the mapping configuration file, specifically:

[0074] Get the field name and field type in the mapping configuration file;

[0075] Generate a table creation statement according to the field name and the field type;

[0076] Create an uninstallation table according to the table creation statement.

[0077] In the first embodiment of the present invention, to create an unload table, it is necessary to first obtain a DDL statement for generating the unload table, and the DDL statement is generated according to the field name and field type specified in the mapping configuration file. The data unloading method of the present invention does not generate temporary files, but directly generates local files, which can effectively improve the data unloading efficiency.

[0078] Step 104: Determine whether there is a complex uninstallation scenario currently according to the association table information in the mapping configuration file.

[0079] Further, in the first embodiment of the present invention, it is determined whether a complex uninstallation scenario currently exists according to the association table information in the mapping configuration file, specifically:

[0080] Get the number of association tables and the file contents of each association table according to the association table information in the mapping configuration file;

[0081] When the number of the association tables is two or more, or the file content contains a subquery, it is determined that a complex uninstallation scenario currently exists;

[0082] When the number of the association table is one and the file content does not contain a subquery, it is determined that no complex uninstallation scenario currently exists.

[0083] In the first embodiment of the present invention, the present invention adopts different data unloading methods for simple data unloading scenarios and complex data unloading scenarios. Therefore, before using the unloading table to unload data, it is first determined whether there is a complex data unloading scenario. The method for determining a complex data unloading scenario is to first obtain the associated table information in the mapping configuration file, determine whether there are multiple associated tables, and whether the associated table contains subqueries. If there is only one associated table and the associated table does not contain subqueries, it is considered that there is no complex data unloading scenario; otherwise, it is considered that there is a complex data unloading scenario.

[0084] Step 105: If a complex unloading scenario currently exists, a temporary table is created according to the associated table information, and the unloading outer table is inserted from the temporary table to perform data unloading.

[0085] Furthermore, in the first embodiment of the present invention, if there is a complex uninstallation scenario, a temporary table is created according to the association table information, specifically:

[0086] A temporary table is created, and the file contents of each of the associated tables are stored in the temporary table.

[0087] In the first embodiment of the present invention, when there is a complex unloading scenario, the data with subquery or multi-table association processing logic is stored in a temporary table, the unloading logic obtained from the mapping configuration file is temporarily stored, and then subsequent SQL splicing, SQL compilation and data unloading are performed. The present invention avoids the problem that the unloading table does not support self-query and multi-table association export by creating a temporary table.

[0088] Step 106: If there is no complex unloading scenario currently, insert the unloading outer table from the physical table where the data to be unloaded is located to unload the data.

[0089] Furthermore, in the first embodiment of the present invention, before inserting the uninstallation table to uninstall data, the method further includes:

[0090] Performing GDS detection on the uninstallation table, and when the uninstallation table passes the GDS detection, using the uninstallation table to perform data uninstallation;

[0091] Wherein, the GDS detection is specifically:

[0092] Acquire a date parameter according to an uninstallation requirement of the data to be uninstalled; wherein the date parameter corresponds to the data to be uninstalled;

[0093] Get the process information corresponding to the GDS service background on the local server;

[0094] Obtain date information from the process information to obtain a GDS specified date;

[0095] Comparing the date parameter with the GDS specified date;

[0096] When the date parameter is consistent with the GDS specified date, it is determined that the uninstallation form has passed the inspection.

[0097] In the first embodiment of the present invention, before data is unloaded, it is necessary to perform a GDS test to check the date corresponding to the unloading to ensure the correctness of the unloading data. Specifically, the process information corresponding to the GDS service background is obtained on the local server, and the date information therein is intercepted to obtain the GDS specified date. The program is used to read the date parameter, which is a program input parameter for unloading data of the specified date. The GDS specified date is compared with the date parameter read by the program. If they are consistent, the test is passed, otherwise it fails. The present invention uses GDS in conjunction with the unloading appearance to achieve high-speed data export.

[0098] In the first embodiment of the present invention, if there is no complex data unloading scenario, data is directly unloaded by inserting data from the physical table into the external table; if there is a complex data unloading scenario, data is unloaded by inserting data from the temporary table into the external table.

[0099] Furthermore, in the first embodiment of the present invention, it also includes:

[0100] After the data is uninstalled, the uninstallation table is deleted.

[0101] In the first embodiment of the present invention, the present invention creates the file format of the unloading table through a mapping configuration file before data unloading, and deletes the unloading table after data unloading is completed, thereby realizing the function of dynamically creating the table.

[0102] As an example of the first embodiment of the present invention, see Figure 2 , is a schematic diagram of the system architecture of an embodiment of the dynamic data unloading system provided by the present invention, wherein CN is a coordination node, DN is a data node, and the dynamic data unloading system is upgraded and optimized based on the traditional external table unloading. The field information required by the external table is controlled through the dynamic unloading mapping configuration file, and the external table required for data unloading is dynamically generated, and then the generated external table is used to unload the corresponding data files. For details, see Figure 3, is a flow chart of another embodiment of the dynamic data unloading method provided by the present invention. The present invention obtains a mapping configuration file from a database, performs a self-check on the mapping configuration file, and then dynamically creates a table based on the information in the mapping configuration file; creates a temporary table for complex data unloading scenarios, and stores the data related to the complex data unloading scenarios obtained from the mapping configuration file in the temporary table; performs a GDS check before unloading the data, and inserts a table into a physical table or a temporary table to complete data unloading for different data unloading scenarios. The present invention dynamically configures complex processing logic through a mapping configuration file, and then implements mapping verification, GDS inspection, SQL splicing, SQL compilation, temporary table creation and data unloading of the complex processing logic through script control, thereby solving the problem that fixed tables cannot meet flexible data unloading, and the problem that traditional tables cannot meet complex data unloading scenarios.

[0103] In the first embodiment of the present invention, the external table format, field order, file type and storage location created by the present invention are diverse and can be dynamically adjusted according to the personalized field requirements of the downstream system, making the data unloading scenario more flexible and easy to operate. The present invention can desensitize or filter the sensitive information fields set by the user during the data unloading process, thereby improving data security and avoiding unnecessary disk space waste in the downstream system.

[0104] In summary, the first embodiment of the present invention provides a method for dynamic data unloading, which includes obtaining a mapping configuration file in a database; performing data verification on the mapping configuration file; creating an unloading table according to the field name and field type in the mapping configuration file after the mapping configuration file passes the data verification; judging whether a complex unloading scenario currently exists according to the associated table information in the mapping configuration file; if a complex unloading scenario currently exists, creating a temporary table according to the associated table information, and inserting the unloading table from the temporary table to unload the data; if no complex unloading scenario currently exists, inserting the unloading table from the physical table where the data to be unloaded is located to unload the data. The present invention realizes flexible data unloading, meets the scenario of complex data unloading, solves the defect that traditional tables can only export single tables, broadens the use of tables, reduces the complexity of data development in data warehouses and downstream systems, and reduces the overall development cycle.

[0105] Example 2

[0106] See also Figure 4 , is a schematic diagram of the structure of an embodiment of a dynamic data unloading device provided by the present invention, the device comprises an acquisition module 201, a verification module 202, an appearance creation module 203, a judgment module 204, a first unloading module 205 and a second unloading module 206;

[0107] The acquisition module 201 is used to acquire a mapping configuration file in a database; wherein the mapping configuration file is generated according to the uninstallation requirements of the data to be uninstalled;

[0108] The verification module 202 is used to perform data verification on the mapping configuration file; wherein the data verification includes primary key uniqueness verification, key field non-empty verification and processing rule standardization verification;

[0109] The appearance creation module 203 is used to create an uninstall appearance according to the field name and field type in the mapping configuration file after the mapping configuration file passes the data verification;

[0110] The judgment module 204 is used to judge whether there is a complex uninstallation scenario at present according to the association table information in the mapping configuration file;

[0111] The first unloading module 205 is used to create a temporary table according to the association table information if there is a complex unloading scenario, and insert the unloading outer table from the temporary table to perform data unloading;

[0112] The second unloading module 206 is used for inserting the unloading outer table from the physical table where the data to be unloaded is located to unload the data if there is no complex unloading scenario currently.

[0113] Furthermore, in the second embodiment of the present invention, the mapping configuration file is specifically:

[0114] According to the unloading requirements of the data to be unloaded, determine the content corresponding to each column name in the mapping configuration file and generate the mapping configuration file;

[0115] The names of the columns in the mapping configuration file include object name, field name, field description, field type, mapping rule, mode name, association table, association type, association condition, filter condition and association table alias.

[0116] Furthermore, in the second embodiment of the present invention, data verification is performed on the mapping configuration file, specifically:

[0117] Performing primary key uniqueness check on each column name in the mapping configuration file, and when there are duplicate column names, determining that the mapping configuration file data check fails;

[0118] Perform non-empty check on key fields of the mapping configuration file. When there are empty values ​​in the field name, field type, mapping rule, association table, association type, association condition, filter condition and association table alias in the mapping configuration file, it is determined that the data check of the mapping configuration file fails.

[0119] The mapping configuration file is checked for processing rule compliance. When a field is missing or there are two or more filter conditions, it is determined that the mapping configuration file data check fails.

[0120] Further, in the second embodiment of the present invention, after the mapping configuration file passes the data verification, an uninstallation table is created according to the field name and field type in the mapping configuration file, specifically:

[0121] Get the field name and field type in the mapping configuration file;

[0122] Generate a table creation statement according to the field name and the field type;

[0123] Create an uninstallation table according to the table creation statement.

[0124] Further, in the second embodiment of the present invention, it is determined whether a complex uninstallation scenario currently exists according to the association table information in the mapping configuration file, specifically:

[0125] Get the number of association tables and the file contents of each association table according to the association table information in the mapping configuration file;

[0126] When the number of the association tables is two or more, or the file content contains a subquery, it is determined that a complex uninstallation scenario currently exists;

[0127] When the number of the association table is one and the file content does not contain a subquery, it is determined that no complex uninstallation scenario currently exists.

[0128] Further, in the second embodiment of the present invention, if there is a complex uninstallation scenario, a temporary table is created according to the association table information, specifically:

[0129] A temporary table is created, and the file contents of each of the associated tables are stored in the temporary table.

[0130] Furthermore, in the second embodiment of the present invention, before inserting the uninstallation table to uninstall data, the method further includes:

[0131] Performing GDS detection on the uninstallation table, and when the uninstallation table passes the GDS detection, using the uninstallation table to perform data uninstallation;

[0132] Wherein, the GDS detection is specifically:

[0133] Acquire a date parameter according to an uninstallation requirement of the data to be uninstalled; wherein the date parameter corresponds to the data to be uninstalled;

[0134] Get the process information corresponding to the GDS service background on the local server;

[0135] Obtain date information from the process information to obtain a GDS specified date;

[0136] Comparing the date parameter with the GDS specified date;

[0137] When the date parameter is consistent with the GDS specified date, it is determined that the uninstallation form has passed the inspection.

[0138] Furthermore, in the second embodiment of the present invention, the mapping configuration file is obtained in the database, specifically:

[0139] Automatically connect to the database and obtain the mapping configuration file in the relational database.

[0140] Furthermore, in the second embodiment of the present invention, the method for dynamically unloading data further includes:

[0141] After the data is uninstalled, the uninstallation table is deleted.

[0142] In summary, the second embodiment of the present invention provides a dynamic data unloading device, which is based on the organic combination between modules, obtains a mapping configuration file in the database; performs data verification on the mapping configuration file; when the mapping configuration file passes the data verification, creates an unloading table according to the field name and field type in the mapping configuration file; determines whether there is a complex unloading scenario at present according to the associated table information in the mapping configuration file; if there is a complex unloading scenario at present, creates a temporary table according to the associated table information, and inserts the unloading table from the temporary table to unload the data; if there is no complex unloading scenario at present, inserts the unloading table from the physical table where the data to be unloaded is located to unload the data. The present invention realizes flexible data unloading, meets the scenario of complex data unloading, solves the defect that traditional tables can only export single tables, broadens the use of tables, reduces the complexity of data development of data warehouses and downstream systems, and reduces the overall development cycle.

[0143] The specific embodiments described above further illustrate the purpose, technical solutions and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. It is particularly pointed out that for those skilled in the art, any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for dynamic data unloading, characterized in that: include: Obtaining a mapping configuration file in a database; wherein the mapping configuration file is generated according to the uninstallation requirements of the data to be uninstalled; Performing data verification on the mapping configuration file; wherein the data verification includes primary key uniqueness verification, key field non-empty verification and processing rule standardization verification; When the mapping configuration file passes the data verification, an uninstallation table is created according to the field name and field type in the mapping configuration file; Determine whether there is a complex uninstallation scenario currently according to the association table information in the mapping configuration file; If there is a complex unloading scenario, a temporary table is created according to the associated table information, and the unloading outer table is inserted from the temporary table to perform data unloading; If there is no complex unloading scenario currently, the unloading outer table is inserted from the physical table where the data to be unloaded is located to perform data unloading.

2. The method for dynamic data unloading according to claim 1, characterized in that: The mapping configuration file is specifically: According to the unloading requirements of the data to be unloaded, determine the content corresponding to each column name in the mapping configuration file and generate the mapping configuration file; The names of the columns in the mapping configuration file include object name, field name, field description, field type, mapping rule, mode name, association table, association type, association condition, filter condition and association table alias.

3. The method for dynamic data unloading according to claim 2, characterized in that: The data verification of the mapping configuration file is specifically performed as follows: Performing primary key uniqueness check on each column name in the mapping configuration file, and when there are duplicate column names, determining that the mapping configuration file data check fails; Perform non-empty check on key fields of the mapping configuration file. When there are empty values ​​in the field name, field type, mapping rule, association table, association type, association condition, filter condition and association table alias in the mapping configuration file, it is determined that the data check of the mapping configuration file fails. The mapping configuration file is checked for processing rule compliance. When a field is missing or there are two or more filter conditions, it is determined that the mapping configuration file data check fails.

4. The method for dynamic data unloading according to claim 3, characterized in that: When the mapping configuration file passes the data verification, the uninstallation table is created according to the field name and field type in the mapping configuration file, specifically: Get the field name and field type in the mapping configuration file; Generate a table creation statement according to the field name and the field type; Create an uninstallation table according to the table creation statement.

5. The method for dynamic data unloading according to claim 3, characterized in that: The determining whether there is a complex uninstallation scenario currently based on the association table information in the mapping configuration file is specifically: Get the number of association tables and the file contents of each association table according to the association table information in the mapping configuration file; When the number of the association tables is two or more, or the file content contains a subquery, it is determined that a complex uninstallation scenario currently exists; When the number of the association table is one and the file content does not contain a subquery, it is determined that no complex uninstallation scenario currently exists.

6. The method for dynamic data unloading according to claim 5, characterized in that: If there is a complex uninstallation scenario, a temporary table is created according to the association table information, specifically: A temporary table is created, and the file contents of each of the associated tables are stored in the temporary table.

7. The method for dynamically unloading data according to claim 5, characterized in that: Before inserting the unloading table to unload data, the method further includes: Performing GDS detection on the uninstallation table, and when the uninstallation table passes the GDS detection, using the uninstallation table to perform data uninstallation; Wherein, the GDS detection is specifically: Acquire a date parameter according to an uninstallation requirement of the data to be uninstalled; wherein the date parameter corresponds to the data to be uninstalled; Get the process information corresponding to the GDS service background on the local server; Obtain date information from the process information to obtain a GDS specified date; Comparing the date parameter with the GDS specified date; When the date parameter is consistent with the GDS specified date, it is determined that the uninstallation form has passed the inspection.

8. The method for dynamic data unloading according to claim 1, characterized in that: The mapping configuration file is obtained from the database, specifically: Automatically connect to the database and obtain the mapping configuration file in the relational database.

9. The method for dynamic data unloading according to claim 1, characterized in that: Also includes: After the data is uninstalled, the uninstallation table is deleted.

10. A data dynamic unloading device, characterized in that: include: An acquisition module, a verification module, an appearance creation module, a judgment module, a first uninstallation module and a second uninstallation module; The acquisition module is used to acquire a mapping configuration file in a database; wherein the mapping configuration file is generated according to the uninstallation requirements of the data to be uninstalled; The verification module is used to perform data verification on the mapping configuration file; wherein the data verification includes primary key uniqueness verification, key field non-empty verification and processing rule standardization verification; The appearance creation module is used to create an uninstall appearance according to the field name and field type in the mapping configuration file after the mapping configuration file passes the data verification; The judging module is used to judge whether there is a complex uninstallation scenario currently according to the association table information in the mapping configuration file; The first unloading module is used to create a temporary table according to the association table information if there is a complex unloading scenario, and insert the unloading outer table from the temporary table to perform data unloading; The second unloading module is used for inserting the unloading outer table into the physical table where the data to be unloaded is located to unload the data if there is no complex unloading scenario currently.

Images