A method, apparatus, device and medium for data import in a database

Abstract

The application provides a data import method, device, equipment and medium in a data warehouse, applied to a data operation layer and a data detail layer of a data warehouse of a target object, the method comprising: the data operation layer loading a latest data table and a historical data table about the target object; the latest data table comprising latest field list data; the historical data table comprising historical field list data and change type data; the data operation layer adjusting the historical field list data and the change type data in the historical data table according to a comparison result of the latest field list data and the historical field list data, obtaining a to-be-imported data table; the data operation layer determining an import mode of importing data in the to-be-imported data table to the data detail layer according to a table type and the change type data of the to-be-imported data table; and the data detail layer generating a management strategy of the target object according to the imported data.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of databases, and in particular, to a method and device for data import in a database, and a device and medium. BACKGROUND

[0002] A data warehouse is a tool for storing data. A data warehouse is a strategic collection of all types of data that support all levels of decision-making processes in an enterprise. It is a single data store created for analytical reporting and decision support purposes. It provides guidance for business process improvement, monitoring of time, cost, quality, and control for enterprises that need business intelligence. Data of an enterprise is stored in a data warehouse, so that the enterprise can manage its operation based on the data in the data warehouse. SUMMARY

[0003] Therefore, the present application aims to provide a method and device for data import in a database, and a device and medium, to solve the problem of how to improve the processing efficiency of data to be stored in a data warehouse in the prior art.

[0004] In a first aspect, the present application provides a method for data import in a data warehouse, applied to a data operation layer and a data detail layer of a data warehouse of a target object, comprising:

[0005] The data operation layer loads a latest data table and a historical data table about the target object; the latest data table includes a latest field list data; the historical data table includes a historical field list data and a change type data;

[0006] The data operation layer adjusts the historical field list data and the change type data in the historical data table according to a comparison result of the latest field list data and the historical field list data, to obtain a to-be-imported data table;

[0007] The data operation layer determines an import mode of data in the to-be-imported data table to the data detail layer according to a table type of the to-be-imported data table and the change type data;

[0008] The data detail layer generates a management strategy of the target object according to the imported data.

[0009] Optionally, the data operation layer adjusts the historical field list data and the change type data in the historical data table according to a comparison result of the latest field list data and the historical field list data, to obtain a to-be-imported data table, comprising:

[0010] In the data operation layer, if the comparison result of the latest field list data and the historical field list data is consistent, the historical data table is determined as the to-be-imported data table;

[0011] In the data operation layer, if the comparison result of the latest field list data and the historical field list data is inconsistent, and there is a deleted field in the latest field list data, the change type data in the historical data table is adjusted to a deletion label, and the deleted field in the historical field list data is deleted to obtain the to-be-imported data table.

[0012] In the data operation layer, if the comparison result of the latest field list data and the historical field list data is inconsistent, and there is an added field in the historical field list data, the change type data in the historical data table is adjusted to an addition label, and the added field is added to the historical field list data to obtain the to-be-imported data table.

[0013] Optionally, in the data operation layer, if the comparison result of the latest field list data and the historical field list data is consistent, the historical data table is determined as the to-be-imported data table, comprising:

[0014] In the data operation layer, if the comparison result of the latest field list data and the historical field list data is consistent, the data in the historical field list data is exchanged, the change type data in the historical data table is adjusted to a variable sequence label, and the historical field list data after the field exchange is determined as the to-be-imported data table.

[0015] Optionally, the data operation layer determines the import mode of the data in the to-be-imported data table into the data detail layer according to the table type and the change type data of the to-be-imported data table, comprising:

[0016] If the table type in the to-be-imported data table is an incremental table, and the change type data is an addition label, the data in the to-be-imported data table is imported into the data detail layer in a full amount manner.

[0017] If the table type in the to-be-imported data table is a full amount table, and the change type data is an addition label, the data in the to-be-imported data table is imported into the data detail layer in a full amount manner.

[0018] If the table type in the to-be-imported data table is an incremental table, and the change type data is not an addition label, the data in the to-be-imported data table is imported into the data detail layer in an incremental manner.

[0019] In a second aspect, the embodiments of the present application provide a device for data import in a data warehouse, which is applied to a data operation layer and a data detail layer of a target object in the data warehouse, and comprises:

[0020] a loading module configured to load, by the data operation layer, a latest data table and a historical data table about the target object; the latest data table comprises latest field list data; and the historical data table comprises historical field list data and change type data;

[0021] an adjusting module configured to adjust, by the data operation layer, the historical field list data and the change type data in the historical data table according to a comparison result of the latest field list data and the historical field list data, to obtain a to-be-imported data table;

[0022] an importing module configured to determine, by the data operation layer, an import mode of data in the to-be-imported data table to the data detail layer according to a table type of the to-be-imported data table and the change type data;

[0023] a management module configured to generate, by the data detail layer, a management strategy of the target object according to the imported data.

[0024] Optionally, the adjusting module comprises:

[0025] a first comparison unit configured to determine, by the data operation layer, the historical data table as the to-be-imported data table if the comparison result of the latest field list data and the historical field list data is consistent;

[0026] a second comparison unit configured to, if the comparison result of the latest field list data and the historical field list data is inconsistent and there is a deleted field in the latest field list data, adjust the change type data in the historical data table to a deletion label, delete the deleted field in the historical field list data, and obtain the to-be-imported data table;

[0027] a third comparison unit configured to, if the comparison result of the latest field list data and the historical field list data is inconsistent and there is an added field in the historical field list data, adjust the change type data in the historical data table to an addition label, and add the added field in the historical field list data, to obtain the to-be-imported data table.

[0028] Optionally, the first comparison unit comprises:

[0029] The adjusting subunit is configured to, in the data operation layer, if the comparison result of the latest field list data and the historical field list data is consistent, perform field switching on the data in the historical field list data, adjust the change type data in the historical data table to a change order label, and determine the historical field list data after the field switching as the to-be-imported data table.

[0030] Optionally, the import module comprises:

[0031] The first import unit is configured to, if the table type in the to-be-imported data table is an incremental table and the change type data is an added label, import the data in the to-be-imported data table to the data detail layer in a full-amount manner.

[0032] The second import unit is configured to, if the table type in the to-be-imported data table is a full-amount table and the change type data is an added label, import the data in the to-be-imported data table to the data detail layer in a full-amount manner.

[0033] The third import unit is configured to, if the table type in the to-be-imported data table is an incremental table and the change type data is a non-added label, import the data in the to-be-imported data table to the data detail layer in an incremental manner.

[0034] In a third aspect, an embodiment of the present application provides a computer device, including a memory, a processor, and a computer program stored in the memory and capable of running on the processor, and the processor implements the steps of the above method when executing the computer program.

[0035] In a fourth aspect, an embodiment of the present application provides a computer readable storage medium, which stores a computer program, and the computer program runs on a processor to execute the steps of the above method.

[0036] The method for data import in a data warehouse provided by the embodiment of the present application firstly loads a latest data table and a historical data table about the target object in the data operation layer; the latest data table comprises latest field list data; the historical data table comprises historical field list data and change type data; secondly, the data operation layer adjusts the historical field list data and the change type data in the historical data table according to the comparison result of the latest field list data and the historical field list data, to obtain a to-be-imported data table; thirdly, the data operation layer determines an import mode of importing the data in the to-be-imported data table to the data detail layer according to the table type and the change type data of the to-be-imported data table; and finally, the data detail layer generates a management strategy of the target object according to the imported data.

[0037] In some embodiments, the latest field list data is compared with the historical field list data, the historical data table is adjusted by the difference between the data, and the adjusted historical data table is stored as a to-be-imported data table in the data detail layer of the data warehouse, without adjusting all the data in the latest data table in terms of data format, only adjusting the data with differences, thereby improving the processing efficiency of the data to be stored in the data warehouse.

[0038] In order to make the above objectives, characteristics and advantages of the present application more obvious and easy to understand, the following preferred embodiments are specifically described below, and the accompanying drawings are described in detail as follows. BRIEF DESCRIPTION OF DRAWINGS

[0039] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments. It should be understood that the following drawings only show some embodiments of the present application, and therefore should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can also be obtained without creative labor on the basis of these drawings.

[0040] Figure 1 A flowchart of a method for data import in a data warehouse provided by an embodiment of the present application is shown in the figure.

[0041] Figure 2 A structural diagram of a device for data import in a data warehouse provided by an embodiment of the present application is shown in the figure.

[0042] Figure 3 A structural diagram of a computer device provided by an embodiment of the present application is shown in the figure. DETAILED DESCRIPTION

[0043] In order to make the objectives, technical solutions and advantages of the embodiments of the present application more clear, the technical solutions in the embodiments of the present application will be described in detail below with the aid of the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments. The components of the embodiments of the present application described and shown in the drawings can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the claimed present application, but only represents selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0044] The embodiments of the present application provide a method for data import in a data warehouse, as shown in the figure, applied to a data operation layer and a data detail layer of a data warehouse of a target object, comprising: Figure 1 The embodiments of the present application provide a method for data import in a data warehouse, as shown in the figure, applied to a data operation layer and a data detail layer of a data warehouse of a target object, comprising:

[0045] S101, the data operation layer loads the latest data table and the historical data table about the target object; the latest data table includes the latest field list data; the historical data table includes the historical field list data and the change type data;

[0046] S102, the data operation layer adjusts the change type data in the historical data table according to the comparison result of the latest field list data and the historical field list data, to obtain a to-be-imported data table;

[0047] S103, the data operation layer determines an import mode of importing data in the to-be-imported data table into the data detail layer according to the table type of the to-be-imported data table and the change type data;

[0048] S104, the data detail layer generates a management strategy of the target object according to the imported data.

[0049] In the above step S101, the target object is an object for storing data in the data warehouse, such as an enterprise, a school, etc. The data operation layer of the data warehouse is located between the source data and the data detail layer, and the data detail layer is used for storing data. The latest data table is a data table composed of source data in a preset time length closest to the current time, and the historical data table is a data table in a second preset time length before the current time. The latest field list data and the historical field list data correspond to the header information of the data table, respectively, and the header information is consistent. The change type data is used to represent whether there is a difference between the data in the latest data table and the data in the historical data table. The change type data includes a change order label, a deletion label and an addition label. The change order label is used to represent that the data in the historical field list data is exchanged. The deletion label is used to represent that there is deleted data in the historical field list data. The addition label is used to represent that there is added data in the historical field list data. Generally, open source tools are used to store source data into the data warehouse for data transmission. In this application, the Sqoop tool is taken as an example to execute the method for importing data in the data warehouse provided in this application.

[0050] Specifically, the data operation layer uses Sqoop to load the source data.

[0051] The Sqoop is used to obtain the latest data table and the historical data table, including:

[0052] A configuration file is loaded, and the library name involved is loaded;

[0053] Records one month ago and today are deleted;

[0054] The latest table information (field list, table name, library name, isused default 0) is inserted into the sqoop_compared table;

[0055] sqoop_config table change type is reset to 0 (no change).

[0056] wherein sqoop_compared is the latest data table and sqoop_config is the historical data table.

[0057] In the above step S102, the data in the data table to be imported is used to import the data details layer in the data warehouse.

[0058] Specifically, different comparison results will make different adjustments to the historical data table, and then obtain the corresponding data table to be imported. The specific processing mode includes the following:

[0059] Step 1021, in the data operation layer, if the comparison result of the latest field list data and the historical field list data is consistent, the historical data table is determined as the data table to be imported.

[0060] Step 1022, in the data operation layer, if the comparison result of the latest field list data and the historical field list data is inconsistent, and there is a deleted field in the latest field list data, the change type data in the historical data table is adjusted to a deletion label, and the deleted field in the historical field list data is deleted, to obtain the data table to be imported.

[0061] Step 1023, in the data operation layer, if the comparison result of the latest field list data and the historical field list data is inconsistent, and there is a new field in the historical field list data, the change type data in the historical data table is adjusted to a new label, and the new field is added to the historical field list data, to obtain the data table to be imported.

[0062] In the above step 1021, if the comparison result of the latest field list data and the historical field list data is consistent, it means that the latest data has not been modified. In order to ensure that the data stored in the data details layer of the data warehouse is uniform in format and meets the specifications of the data warehouse, the data to be stored in the data details layer also needs to be adjusted in field order.

[0063] That is, step 1021 includes:

[0064] Step 10211, in the data operation layer, if the comparison result of the latest field list data and the historical field list data is consistent, the data in the historical field list data is exchanged in field, the change type data in the historical data table is adjusted to a variable order label, and the historical field list data after field exchange is determined as the data table to be imported.

[0065] In step 10211, specific embodiments include:

[0066] Query the field list of the table (sqoop_config in the field list of sqoop_select_list) and compare it with the latest field list;

[0067] If the comparison result is consistent, the sqoop_config table change type remains 0, and the update field and the delete field are empty;

[0068] And if the new and deleted fields are empty, it is the field exchange order, and the change type in the sqoop_config table is set to 3 (change order);

[0069] Finally, the sqoop_config table after the field exchange order is used as the data table to be imported.

[0070] In step 1022, specific embodiments include:

[0071] If the comparison result is inconsistent, set sqoop_compared table isused to 1 (send a reminder email);

[0072] Traverse the new and old field lists, and if a specific deleted field is found;

[0073] In the sqoop_config table, set the change type to 2 (with deletion);

[0074] In the sqoop_config table, replace the deleted field in sqoop_select_list (the current field list) with null as 'field name';

[0075] Update the delete field to the delete field;

[0076] Finally, the sqoop_config table after deleting the field is used as the data table to be imported.

[0077] In step 1023, specific embodiments include:

[0078] If the comparison result is inconsistent, set sqoop_compared table isused to 1 (send a reminder email);

[0079] Traverse the new and old field lists, and if a specific added field is found;

[0080] In the sqoop_config table, set the change type to 1 (with addition);

[0081] In sqoop_select_list (current field list) and dwd_select_list (select list of SQL from ods to dwd) of sqoop_config table, splice the new field, update the new field to the new field;

[0082] Update the special field in sqoop: traverse the update field list, query the data type, and add 'field name' = String to special_column in sqoop_config table for the field with data type with '[]', 'json', 'geometry';

[0083] Finally, the sqoop_config table after adding the new field is used as the data table to be imported.

[0084] After determining the data table to be imported, the data table to be imported needs to be imported into the data detail layer, which completes the storage of source data to the data warehouse. However, according to the different comparison results of the latest field list data and the historical field list data, the data table to be imported of different table types can be determined, and then different import methods are used to store the data in the data table to be imported. Specifically, step 103 includes the following import methods:

[0085] Step 1031, if the table type of the data table to be imported is an incremental table, and the change type data is a new label, the data in the data table to be imported is imported into the data detail layer in a full amount manner;

[0086] Step 1032, if the table type of the data table to be imported is a full amount table, and the change type data is a new label, the data in the data table to be imported is imported into the data detail layer in a full amount manner;

[0087] Step 1033, if the table type of the data table to be imported is an incremental table, and the change type data is a non-new label, the data in the data table to be imported is imported into the data detail layer in an incremental manner.

[0088] When steps 1031 to 1033 are executed, the following instructions are used when the data in the data table to be imported is imported into the data detail layer using the Sqoop tool:

[0089] Pass in parameters, -t create / no, -o library name (separated by commas, if there is a table with isused = 2, this parameter is invalid);

[0090] Load configuration file, load the table name involved;

[0091] Query sqoop_select_list, where condition, splice into SQL;

[0092] If the parameter-t create, splice sqoop for --create-hive-table (the default is false, if the target table already exists, then the creation task will fail);

[0093] Query field change type, full table or incremental table.

[0094] In the above step 1031, specific embodiments include:

[0095] If the field is added, it is also an incremental table;

[0096] Rename the original table (change the incremental table to the full table);

[0097] Splice SQL (limit 1) and pull the latest table structure (sqoop create hive table);

[0098] Query the new field, set the new field to null, splice SQL, import the original table data into the data detail layer of the latest structure table in full quantity, and delete the original table; tabletype is empty.

[0099] Through the above table name change, the data stored in the data warehouse is all full table data, and the data in the data warehouse is format unified, improving the query efficiency and utilization of the data by the staff later.

[0100] In the above step 1032, specific embodiments include:

[0101] If the field is added, it is not an incremental table;

[0102] Delete table; tabletype = "--create-hive-table", sqoop new table synchronizes full quantity data;

[0103] And import the full table into the data detail layer in the data warehouse through the overwrite instruction.

[0104] In the above step 1033, specific embodiments include:

[0105] If the field is not added (deleted, changed order), it is an incremental table: tabletype is empty;

[0106] Other tabletype = "--hive-overwrite";

[0107] Restore the null of the deleted field in sqoop_select_list, to facilitate the judgment tomorrow (next time).

[0108] The data table to be imported is imported from the ODS layer (data operation layer) to the DWD layer (data detail layer), and the specific import method can be simply summarized by the following steps:

[0109] Incoming parameters, -t create / no, -o library name (separated by commas, if there is a table with isused = 2, this parameter is invalid);

[0110] Query whether the field has changed, if new, delete the table first;

[0111] Query the de-duplication grouping field, the de-duplication sorting field, query the full table or the incremental table, and add back quotes to the floor keyword;

[0112] If it is a full table, it is directly overwritten and imported, and if it is an incremental table, it is imported after de-duplication and sorting.

[0113] In the above step S105, the data stored in the data detail layer can be used by the staff to make a management strategy for the target object according to the data. The management strategy includes benefit analysis, cost control, etc.

[0114] An apparatus for data import in a data warehouse is provided, as shown in Figure 2 applied to a data operation layer and a data detail layer of a data warehouse of a target object, comprising:

[0115] The loading module 201 is configured to load the latest data table and the historical data table about the target object in the data operation layer; the latest data table comprises latest field list data; and the historical data table comprises historical field list data and change type data.

[0116] The adjusting module 202 is configured to adjust the historical field list data and the change type data in the historical data table according to a comparison result of the latest field list data and the historical field list data, to obtain a data table to be imported.

[0117] The import module 203 is configured to determine an import mode of importing data in the data table to be imported to the data detail layer according to a table type and the change type data of the data table to be imported.

[0118] The management module 204 is configured to generate a management strategy of the target object according to the imported data in the data detail layer.

[0119] Optionally, the adjusting module comprises:

[0120] The first comparison unit is configured to determine the historical data table as the to-be-imported data table if the comparison result of the latest field list data and the historical field list data is consistent in the data operation layer.

[0121] The second comparison unit is configured to adjust the change type data in the historical data table to a deletion label, delete the deletion field in the historical field list data, and obtain the to-be-imported data table if the comparison result of the latest field list data and the historical field list data is inconsistent and there is a deletion field in the latest field list data in the data operation layer.

[0122] The third comparison unit is configured to adjust the change type data in the historical data table to an addition label, and add the addition field in the historical field list data to obtain the to-be-imported data table if the comparison result of the latest field list data and the historical field list data is inconsistent and there is an addition field in the historical field list data in the data operation layer.

[0123] Optionally, the first comparison unit comprises:

[0124] The adjusting subunit is configured to perform field switching on the data in the historical field list data, adjust the change type data in the historical data table to a change order label, and determine the historical field list data after the field switching as the to-be-imported data table if the comparison result of the latest field list data and the historical field list data is consistent in the data operation layer.

[0125] Optionally, the import module comprises:

[0126] The first import unit is configured to import the data in the to-be-imported data table into the data detail layer in a full-amount manner if the table type in the to-be-imported data table is an incremental table and the change type data is an addition label.

[0127] The second import unit is configured to import the data in the to-be-imported data table into the data detail layer in a full-amount manner if the table type in the to-be-imported data table is a full-amount table and the change type data is an addition label.

[0128] The third import unit is configured to import the data in the to-be-imported data table into the data detail layer in an incremental manner if the table type in the to-be-imported data table is an incremental table and the change type data is not an addition label.

[0129] Corresponding to the method for data import in the data warehouse in Figure 1 The embodiments of the present application also provide a computer device 300, which comprises the data import device 100. Figure 3 As shown, the device includes a memory 301, a processor 302, and a computer program stored in the memory 301 and executable on the processor 302, wherein the processor 302 implements the method for importing data into the data warehouse when executing the computer program.

[0130] Specifically, the above-mentioned memory 301 and processor 302 can be general-purpose memory and processor, which are not specifically limited here. When the processor 302 runs the computer program stored in the memory 301, it can execute the above-mentioned method of importing data into the data warehouse, solving the problem in the existing technology of how to improve the processing efficiency of data stored in the data warehouse.

[0131] Corresponding to Figure 1 A method for importing data into a data warehouse in the embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is run by a processor, the steps of the method for importing data into the data warehouse are executed, solving the problem of how to improve the processing efficiency of data stored in the data warehouse. The present application compares the latest field list data with the historical field list data, and uses the differences between the data to adjust the data in the historical data table, and then stores the adjusted historical data table as the data table to be imported into the data detail layer of the data warehouse. There is no need to adjust the data format of all data in the latest data table, only the data with differences is adjusted, thereby improving the processing efficiency of data stored in the data warehouse.

[0132] Specifically, the storage medium can be a general storage medium, such as a mobile disk, hard disk, etc. When the computer program on the storage medium is run, it can execute the above-mentioned method of importing data into the data warehouse, solving the problem of improving the effectiveness of data stored in the data warehouse in the existing technology.

[0133] In the embodiments provided in this application, it should be understood that the disclosed methods and devices can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interface, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0134] The units described as separate components may or may not be physically separate, and the components displayed as units may or may not be physical units, i.e. may be located in one place, or may be distributed on multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment scheme.

[0135] In addition, the functional units in the embodiments provided in the present application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit.

[0136] The functions, if realized in the form of software functional units and sold or used as independent products, can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application can be embodied in the form of software products, and the computer software products are stored in a storage medium, including a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0137] It should be noted that: similar reference numbers and letters represent similar items in the following drawings, therefore, once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings, in addition, the terms "first", "second", "third" and the like are only used to distinguish description, and cannot be understood as indicating or implying relative importance.

[0138] Finally, it should be noted that: the above-described embodiments are only specific embodiments of the present application, used to illustrate the technical solutions of the present application, and not to limit the same, the protection scope of the present application is not limited thereto, although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any skilled person in the art can modify or easily think of changes to the technical solutions described in the foregoing embodiments within the technical scope disclosed by the present application, or make equivalent replacement to part of the technical features; and these modifications, changes or replacements do not make the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application. All should be covered in the protection scope of the present application. Therefore, the protection scope of the present application should be limited by the protection scope of the claims.

Claims

1. A method for importing data into a data warehouse, characterized in that: The data operation layer and data detail layer of the data warehouse applied to the target object include: The data operation layer loads the latest data table and historical data table of the target object; the latest data table includes the latest field list data; the historical data table includes historical field list data and change type data; wherein the latest data table is a data table composed of source data within a preset time period closest to the current moment, and the historical data table is a data table within a second preset time period before the current moment in the data warehouse; the header information of the data tables corresponding to the latest field list data and the historical field list data are consistent; The data operation layer adjusts the historical field list data and the change type data in the historical data table according to the comparison result of the latest field list data and the historical field list data to obtain the data table to be imported; The data operation layer determines an import method for importing the data in the data table to be imported into the data detail layer according to the table type and change type data of the data table to be imported; The data detail layer generates a management policy for the target object based on the imported data.

2. The method according to claim 1, characterized in that The data operation layer adjusts the historical field list data and the change type data in the historical data table according to the comparison result of the latest field list data and the historical field list data to obtain the data table to be imported, including: In the data operation layer, if the comparison result of the latest field list data and the historical field list data is consistent, the historical data table is determined as the data table to be imported; In the data operation layer, if the comparison result between the latest field list data and the historical field list data is inconsistent, and there is a deleted field in the latest field list data, the change type data in the historical data table is adjusted to a deletion label, and the deleted field in the historical field list data is deleted to obtain the data table to be imported; In the data operation layer, if the comparison results of the latest field list data and the historical field list data are inconsistent, and there are new fields in the historical field list data, the change type data in the historical data table is adjusted to a new label, and the new fields are added to the historical field list data to obtain the data table to be imported.

3. The method according to claim 2, characterized in that In the data operation layer, if the comparison result of the latest field list data and the historical field list data is consistent, then determining the historical data table as the data table to be imported includes: In the data operation layer, if the comparison result of the latest field list data is consistent with the historical field list data, the data in the historical field list data is swapped, and the change type data in the historical data table is adjusted to a variable order label, and the historical field list data after the field swap is determined as the data table to be imported.

4. The method according to claim 1, wherein The data operation layer determines, based on the table type and change type data of the data table to be imported, an import method for importing the data in the data table to be imported into the data detail layer, including: If the table type in the data table to be imported is an incremental table, and the change type data is a new tag, then all the data in the data table to be imported is imported into the data detail layer; If the table type in the data table to be imported is a full table, and the change type data is a new tag, the data in the data table to be imported is imported into the data detail layer in full; If the table type in the data table to be imported is an incremental table, and the change type data is a non-newly added tag, the data in the data table to be imported is imported into the data detail layer in an incremental manner.

5. A device for importing data into a data warehouse, characterized in that: The data operation layer and data detail layer of the data warehouse applied to the target object include: A loading module is used for the data operation layer to load the latest data table and historical data table of the target object; the latest data table includes the latest field list data; the historical data table includes historical field list data and change type data; wherein the latest data table is a data table composed of source data within a preset time period closest to the current moment, and the historical data table is a data table within a second preset time period before the current moment in the data warehouse; the header information of the data tables corresponding to the latest field list data and the historical field list data are consistent; An adjustment module, configured for the data operation layer to adjust the historical field list data and the change type data in the historical data table according to a comparison result between the latest field list data and the historical field list data, to obtain a data table to be imported; An import module, configured for the data operation layer to determine an import method for importing the data in the data table to be imported into the data detail layer according to the table type and change type data of the data table to be imported; A management module is used for the data detail layer to generate a management strategy for the target object based on the imported data.

6. The device according to claim 5, characterized in that The adjustment module includes: a first comparison unit configured to, in the data operation layer, determine the historical data table as the data table to be imported if a comparison result between the latest field list data and the historical field list data is consistent; a second comparison unit configured to, in the data operation layer, adjust the change type data in the historical data table to a deletion label and delete the deleted fields in the historical field list data if the comparison result between the latest field list data and the historical field list data is inconsistent, and the latest field list data contains deleted fields, thereby obtaining the data table to be imported; The third comparison unit is used to adjust the change type data in the historical data table to a new label and add new fields to the historical field list data in the data operation layer, if the comparison results of the latest field list data and the historical field list data are inconsistent, and there are new fields in the historical field list data, to obtain the data table to be imported.

7. The device according to claim 6, characterized in that The first comparison unit includes: An adjustment subunit is used to, in the data operation layer, perform field swaps on the data in the historical field list data if the comparison result of the latest field list data is consistent with the historical field list data, and adjust the change type data in the historical data table to a variable order label, and determine the historical field list data after the field swap as the data table to be imported.

8. The device according to claim 5, characterized in that The import module includes: A first importing unit is configured to import all the data in the data table to be imported into the data detail layer if the table type in the data table to be imported is an incremental table and the change type data is a newly added tag; a second importing unit configured to import the data in the data table to be imported into the data detail layer in full if the table type in the data table to be imported is a full table and the change type data is a newly added tag; The third importing unit is configured to import the data in the data table to be imported into the data detail layer in an incremental manner if the table type in the data table to be imported is an incremental table and the change type data is a non-newly added tag.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 4 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 4 are executed.

Images