A data extraction method, system and terminal device

Abstract

The present invention is applicable to the field of computer technology, and provides a data extraction method, system and terminal device, including: comparing the data source table structure of the current version of the extraction database with the data source table structure of the previous version of the extraction database, judging whether the data source table structure of the extraction database has changed; if the data source table structure of the extraction database has changed, generating a script file for modifying the target table structure according to the changed content; modifying the target table structure according to the script file for modifying the target table structure, and generating an extraction script file according to the modified target table structure; extracting data from the extraction database according to the extraction script file. The latest extraction script file can be generated according to the table structure of the latest target table to extract data, without manually modifying the extraction script file, and automatically extracting data according to the change of the data source table structure, thereby reducing manual intervention and improving the efficiency and accuracy of data extraction.

Description

Technical Field

[0001] The present invention belongs to the field of computer technology, and in particular relates to a data extraction method, system and terminal equipment. Background Art

[0002] Database systems are widely used in data-intensive fields such as communications and finance. For data-intensive fields, data access, storage, and upgrades are particularly important. Therefore, it is often necessary to extract data from multiple database systems for data analysis. Data extraction is the process of extracting data from a database. In practical applications, relational databases are more commonly used.

[0003] In a relational database, a database table is a collection of two-dimensional arrays that represent and store the relationship between data objects. When a database table changes, it is necessary to generate sqoop derivative scripts in batches at one time to obtain the modified database table structure. After obtaining the modification information of the database table structure, it is necessary to manually modify the script file for extracting data according to the modification information. In addition, when there are many data tables that need to extract data, modifying the script file for extracting data requires a lot of manpower and time, and is prone to errors. When adding a new table for extracting data or modifying some fields of some data tables, it also takes a lot of manpower and time to write the script file for adding the database table structure and the script file for modifying the database table structure.

[0004] In summary, the current data extraction work has problems such as high time cost, high labor cost and low work efficiency. Summary of the invention

[0005] In view of this, the embodiments of the present invention provide a data extraction method, system and terminal device to solve the problems of high time cost, high labor cost and low work efficiency in the current data extraction work.

[0006] A first aspect of the present invention provides a data extraction method, comprising:

[0007] Compare the data source table structure of the current version of the extraction database with the data source table structure of the previous version of the extraction database to determine whether the data source table structure of the extraction database has changed;

[0008] If the data source table structure of the extraction database changes, a script file for modifying the target table structure is generated according to the changed content;

[0009] Modify the target table structure according to the script file for modifying the target table structure, and generate a number extraction script file according to the modified target table structure;

[0010] Data is extracted from the extraction database according to the extraction script file.

[0011] A second aspect of the present invention provides a data extraction system, comprising:

[0012] A judgment module, used to compare the data source table structure of the current version of the extraction database with the data source table structure of the previous version of the extraction database, and judge whether the data source table structure of the extraction database has changed;

[0013] A generating module, for generating a script file for modifying the target table structure according to the changed content if the data source table structure of the extraction database changes;

[0014] A modification module, used for modifying the target table structure according to the script file for modifying the target table structure, and generating a number extraction script file according to the modified target table structure;

[0015] An extraction module is used to extract data from the extraction database according to the extraction script file.

[0016] A third aspect of the present invention provides a terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the following steps when executing the computer program:

[0017] Compare the data source table structure of the current version of the extraction database with the data source table structure of the previous version of the extraction database to determine whether the data source table structure of the extraction database has changed;

[0018] If the data source table structure of the extraction database changes, a script file for modifying the target table structure is generated according to the changed content;

[0019] Modify the target table structure according to the script file for modifying the target table structure, and generate a number extraction script file according to the modified target table structure;

[0020] Data is extracted from the extraction database according to the extraction script file.

[0021] A fourth aspect of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0022] Compare the data source table structure of the current version of the extraction database with the data source table structure of the previous version of the extraction database to determine whether the data source table structure of the extraction database has changed;

[0023] If the data source table structure of the extraction database changes, a script file for modifying the target table structure is generated according to the changed content;

[0024] Modify the target table structure according to the script file for modifying the target table structure, and generate a number extraction script file according to the modified target table structure;

[0025] Data is extracted from the extraction database according to the extraction script file.

[0026] The present invention provides a data extraction method, system and terminal device, which generate a script file corresponding to the modified target table structure by comparing the changes in the data source table structure, modify the table structure of the target table, and then generate the latest extraction script file according to the latest table structure of the target table to extract data. There is no need to manually modify the extraction script file, and data can be automatically extracted according to the changes in the data source table structure, thereby reducing manual intervention, improving the efficiency and accuracy of data extraction, and effectively solving the problems of high time cost, high labor cost and low work efficiency in the current data extraction work. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0028] Figure 1 This is a schematic diagram of an implementation flow of a data extraction method provided in Embodiment 1 of the present invention;

[0029] Figure 2 It is a schematic diagram of the implementation flow of step S101 of the corresponding embodiment 1 provided by the second embodiment of the present invention;

[0030] Figure 3 It is a schematic diagram of the implementation flow of step S104 of the corresponding embodiment 1 provided in the third embodiment of the present invention;

[0031] Figure 4 is a structural schematic diagram of a data extraction system provided by Embodiment 4 of the present invention;

[0032] Figure 5 is a schematic diagram of the structure of the judgment module 101 in the corresponding embodiment 4 provided in the fifth embodiment of the present invention;

[0033] Figure 6 is a schematic diagram of the structure of the extraction module 104 in the corresponding embodiment 4 provided by the sixth embodiment of the present invention;

[0034] Figure 7 It is a schematic diagram of a terminal device provided in Embodiment 7 of the present invention. DETAILED DESCRIPTION

[0035] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present invention. However, it should be clear to those skilled in the art that the present invention may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, systems, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present invention.

[0036] In order to solve the problems of high time cost, high labor cost and low work efficiency in the current data extraction work, the embodiments of the present invention provide a data extraction method, system and terminal device, which compares the changes in the data source table structure, generates a script file for modifying the target table structure, modifies the table structure of the target table, and then generates the latest extraction script file according to the latest table structure of the target table to extract data, without manually modifying the extraction script file, and can automatically extract data according to the changes in the data source table structure, thereby reducing manual intervention, improving the efficiency and accuracy of data extraction, and effectively solving the problems of high time cost, high labor cost and low work efficiency in the current data extraction work.

[0037] In order to illustrate the technical solution of the present invention, a specific embodiment is provided below for illustration.

[0038] Embodiment 1:

[0039] like Figure 1 As shown, this embodiment provides a data extraction method, which specifically includes:

[0040] Step S101: comparing the data source table structure of the current version of the extraction database with the data source table structure of the previous version of the extraction database to determine whether the data source table structure of the extraction database has changed.

[0041] It should be noted that the table structure consists of three parts: the table name, the fields in the table, and the records in the table. The table structure describes the framework of a database, defines the number of fields that make up a database, the name of each field, the data type of each field, the length of each field, and other information. Before establishing a database, you need to pre-design its data source table structure and store the relevant information of the table structure in the metadata of the database.

[0042] In specific applications, the extraction database is the object for data extraction, that is, the data source, and therefore, the table structure of the extraction database is the data source table structure. By comparing the data source table structure of the current version of the extraction database with the data source table structure of the previous version of the extraction database, it is determined whether the data source table structure of the extraction database has changed. Specifically, by comparing whether there is a difference between the field information of the current version of the data source table structure and the field information of the previous version of the data source table structure, it is determined whether the data source table structure has changed. If there is a difference between the field information of the current version of the data source table structure and the field information of the previous version of the data source table structure, the data source table structure has changed. If there is no difference between the field information of the current version of the data source table structure and the field information of the previous version of the data source table structure, the data source table structure has not changed.

[0043] Step S102: if the data source table structure of the extraction database changes, a script file for modifying the target table structure is generated according to the changed content.

[0044] It should be noted that the data extraction process is to extract the data from the extraction database and save it in the target database. The target table structure is the table structure of the target database. The target table structure needs to be adaptively modified according to changes in the data source table structure.

[0045] In specific applications, a script file for modifying the target table structure is generated by analyzing the changes in the data source table structure. SQL statements can be spliced ​​to form a standard data manipulation statement (DML statement), and the spliced ​​DML statement is then written into a new script file, which is the script file for modifying the target table structure.

[0046] Step S103: modifying the target table structure according to the script file for modifying the target table structure, and generating a number extraction script file according to the modified target table structure.

[0047] In a specific application, after obtaining a script file for modifying the target table structure, the target table structure is modified according to the script file, and then the latest extraction script file is generated according to the latest target table structure. Specifically, by querying the target table structure of the current version of the target database (the latest target table structure), the extraction commands of the corresponding fields are spliced ​​according to the target table structure, and the spliced ​​extraction commands are written into a script file, that is, the script file is an extraction script file for adaptive data source table structure changes.

[0048] Step S104: extracting data from the extraction database according to the extraction script file.

[0049] In specific applications, a scheduled task is set up through the task scheduling system to schedule the data extraction task of the extraction database, so that it can automatically run the extraction script file on a regular basis every day, extract the data that needs to be extracted from the extraction database into the target database, and thus realize automatic data extraction.

[0050] In one embodiment, the data extraction method further includes the following steps before step S101:

[0051] Step S105: Identify whether there is an XML file storing the previous version of the data source table structure.

[0052] Step S106: If the XML file does not exist, connect to the database through the database connection interface, and search and obtain the corresponding data source table structure through the database.

[0053] Step S107: Create an XML file and save the data source table structure in the XML file.

[0054] In a specific application, in order to determine whether the data source table structure has changed, it is necessary to obtain the previous version of the data source table structure. Usually, for the extraction database that extracts data for the first time, there is no XML file that saves the previous version of the data source table structure. Therefore, it is necessary to first identify whether the XML file storing the data source table structure exists. If it does not exist, use the jdbc connection method to connect to the extraction database source to query and obtain the table structure of the database to be extracted, and then store the obtained table structure, that is, create an XML file and save the data source table structure in the XML file.

[0055] It should be noted that for the extraction database that extracts data for the first time, the data source table structure is the same as the target source table structure. When extracting data, the data of the entire extraction database is fully extracted. In the case where there is an XML file that stores the data source table structure, the jdbc connection method is used to connect to the extraction database to query and obtain the current version of the data source table structure, and then compare it with the previous version of the data source table structure stored in the XML file, and then adaptively generate the extraction script file according to the changes in the data source table structure, and perform incremental extraction on the extraction database.

[0056] The data extraction method provided in the present embodiment generates a script file for modifying the target table structure by comparing the changes in the data source table structure, and then generates the latest extraction script file according to the latest target table structure to extract data. There is no need to manually modify the extraction script file, and data can be automatically extracted according to the changes in the data source table structure, thereby reducing manual intervention, improving the efficiency and accuracy of data extraction, and effectively solving the problems of high time cost, high labor cost and low work efficiency in the current data extraction work.

[0057] Embodiment 2:

[0058] like Figure 2 As shown, in this embodiment, step S101 in embodiment 1 specifically includes:

[0059] Step S201: query the metadata of the abstraction database through structured query language, and obtain the data source table structure of the current version according to the metadata.

[0060] In specific applications, metadata is data about data used to describe data, mainly information describing data properties, supporting functions such as indicating storage location, historical data, resource search, file records, etc. In this embodiment, metadata is stored in an Execl file, including field information such as table name, group name, library name, source table structure information, and incremental timestamp. The metadata of the extraction database can be queried through structured query language to obtain its source table structure information, and the data source table structure of the current version of the extraction database can be obtained based on the source table structure information.

[0061] Step S202: obtaining the data source table structure of the previous version of the sampling database stored in the XML file.

[0062] In a specific application, the data source table structure of the previous version refers to the data source table structure stored in an XML file. By accessing the XML file, the data source table structure of the previous version of the sampling database can be obtained.

[0063] Step S203: determining whether there are changed fields between the data source table structure of the current version and the data source table structure of the previous version, wherein the changed fields include added fields, deleted fields, and modified fields.

[0064] In specific applications, by comparing the fields of the current version of the data source table structure with the fields of the previous version of the data source table structure, it is determined whether there are changed fields. That is, compared with the previous version of the data source table structure, whether the current version of the data source table structure has added fields, deleted fields, and whether the attributes of the fields (field name, field length, field data type, etc.) have been modified.

[0065] Step S204: If there are changed fields between the data source table structure of the current version and the data source table structure of the previous version, the data source table structure of the extraction database is changed.

[0066] In a specific application, if the data source table structure of the current version has added fields, deleted fields, or modified field properties compared with the data source structure of the previous version, it indicates that the data source table structure of the extraction database has changed.

[0067] Embodiment three:

[0068] like Figure 3 As shown, in this embodiment, step S104 in embodiment 1 specifically includes:

[0069] Step S301: establishing a sampling task, wherein the sampling task includes an execution mode of the sampling task, and the execution mode includes full sampling and incremental sampling.

[0070] In specific applications, a sampling task is established, and the sampling task of the sampling database is scheduled through the task scheduling system. The task scheduling system is used to automatically run the sampling script file every day, so that the data to be extracted from the sampling database is extracted into the target database, thereby realizing automatic data extraction.

[0071] In a specific application, the above-mentioned extraction task includes an execution mode of the extraction task, and the above-mentioned execution mode includes full extraction and incremental extraction.

[0072] In a specific application, a full extraction method is used to extract data from a sampling database that extracts data for the first time, and an incremental extraction method is used to extract data from a sampling database that extracts data not for the first time.

[0073] Step S302: If the execution mode is full extraction, the extraction script file is run to extract all data in the extraction database.

[0074] In a specific application, if the corresponding extraction database extracts data for the first time, the data source table structure is the same as the target table structure, so the generated extraction script file extracts all the data in the extraction database.

[0075] Step S303: If the execution mode is incremental extraction, the extracted data content is determined according to the incremental timestamp of the metadata of the extraction database, and the extraction script file is run according to the extracted data content.

[0076] In specific applications, for a database that is not extracted for the first time, only the changed data needs to be extracted during data extraction. At this time, the target table structure has been modified to adapt to the changes in the data source table structure, and the generated extraction script file has also adapted to the changes in the data source table structure. Therefore, the generated extraction script file is an incremental extraction script file.

[0077] In specific applications, the incremental timestamp can be used as the basis for extracting data to determine the data content that needs to be extracted.

[0078] In specific applications, the incremental timestamp (updated_date) is a field that represents time. This field is stored in the data source table structure of the extraction database. When each piece of data in the extraction database changes (including addition and modification), the database system will modify the value of the updated_date field. Therefore, the incremental timestamp is used to determine the newly added or modified records, and then the newly added or modified data is extracted, that is, incremental extraction.

[0079] In one embodiment, before step S302, the following steps are also included:

[0080] Step S303: Determine whether the data table structure of the current version of the extraction database has an incremental timestamp.

[0081] Step S304: If the data table structure of the current version of the extraction database does not have an incremental timestamp, a unique index field is established, and the extracted data content is determined by the unique index field.

[0082] In a specific application, when there is no incremental timestamp in the sampling database, a unique index field is established and data is extracted through the unique index. A unique index refers to a field that does not allow the same index value. All values ​​of the field are unique in the table and can uniquely identify a record of data in the sampling database. After the extracted data content is determined by the unique index field, data extraction is performed according to the extracted data content.

[0083] In specific applications, for the extracted data table without incremental timestamp fields, the extracted data content is determined according to the unique index field of the source table. The specific operations for data extraction are as follows:

[0084] (1) Import all the data in the sampling database into the intermediate table;

[0085] (2) Obtain all the data in the target database and import them into the target table (the data in the target table is the data of the previous day at this time), compare the data in the intermediate table with the data in the target table, and perform an associated search using a field with a unique index;

[0086] (3) If the value stored in the field with a unique index in the intermediate table is not in the target table, it is new data and can be inserted into the target table;

[0087] (4) If the value stored in the field with a unique index in the intermediate table exists in the target table, then compare the values ​​of all fields in the two tables that are not unique indexes one by one. If the values ​​of any field are different, then mark this field of the record as modified and insert the modified data into the target table.

[0088] Embodiment 4:

[0089] like Figure 4 As shown, this embodiment provides a data extraction system 100 for executing the method steps in the first embodiment, which includes a judgment module 101, a generation module 102, a modification module 103 and an extraction module 104.

[0090] The judgment module 101 is used to compare the data source table structure of the current version of the extraction database with the data source table structure of the previous version of the extraction database to judge whether the data source table structure of the extraction database has changed.

[0091] The generating module 102 is used to generate a script file for modifying the target table structure according to the changed content if the data source table structure of the extraction database changes.

[0092] The modification module 103 is used to modify the target table structure according to the script file for modifying the target table structure, and generate a number extraction script file according to the modified target table structure.

[0093] The extraction module 104 is used to extract data from the extraction database according to the extraction script file.

[0094] In one embodiment, the data extraction system 100 further includes an identification module, a search module and a storage module.

[0095] The identification module is used to identify whether there is an XML file storing the previous version of the data source table structure.

[0096] The search module is used to connect to the database through the database connection interface if the XML file does not exist, and to search and obtain the corresponding data source table structure through the database.

[0097] The saving module is used to create an XML file and save the data source table structure in the XML file.

[0098] It should be noted that the data extraction system provided in the embodiment of the present invention is Figure 1 The method embodiments shown are based on the same concept, and the technical effects brought by them are the same as those of the present invention. Figure 1 The method embodiments shown are the same, and the specific contents can be found in the present invention Figure 1 The description in the illustrated method embodiment will not be repeated here.

[0099] Therefore, a data extraction system provided in this embodiment can also modify the table structure of the target table by comparing the changes in the data source table structure and generating a script file corresponding to the modified target table structure, and then generate the latest extraction script file according to the latest table structure of the target table to extract data. There is no need to manually modify the extraction script file, and data can be automatically extracted according to the changes in the data source table structure, which reduces manual intervention, improves the efficiency and accuracy of data extraction, and effectively solves the problems of high time cost, high labor cost and low work efficiency in the current data extraction work.

[0100] Embodiment five:

[0101] like Figure 5 As shown, in this embodiment, the judgment module 101 in the fourth embodiment includes a Figure 2 The structure of the method steps in the corresponding embodiment includes a structure query unit 201 , an acquisition unit 202 and a judgment unit 203 .

[0102] The structure query unit 201 is used to query the metadata of the abstract database through the structured query language, and obtain the data source table structure of the current version according to the metadata.

[0103] The acquisition unit 202 is used to acquire the data source table structure of the previous version of the sampling database stored in the XML file.

[0104] The judgment unit 203 is used to judge whether there are changed fields between the data source table structure of the current version and the data source table structure of the previous version, and the changed fields include added fields, deleted fields and modified fields; if there are changed fields between the data source table structure of the current version and the data source table structure of the previous version, the data source table structure of the sampling database changes.

[0105] Embodiment six:

[0106] like Figure 6 As shown, in this embodiment, the extraction module 104 in the fourth embodiment includes a module for executing Figure 3The structure of the method steps in the corresponding embodiment includes a task establishment unit 301 , a full amount extraction unit 302 and an incremental extraction unit 303 .

[0107] The task establishing unit 301 is used to establish a sampling task, wherein the sampling task includes an execution mode of the sampling task, and the execution mode includes full sampling and incremental sampling.

[0108] The full extraction unit 302 is used to run the extraction script file to extract all the data in the extraction database if the execution mode is full extraction.

[0109] The incremental extraction unit 303 is used to determine the extracted data content according to the incremental timestamp of the metadata of the extraction database if the execution mode is incremental extraction, and run the extraction script file according to the extracted data content.

[0110] In one embodiment, the extraction module 104 further includes an incremental timestamp unit and an index field unit.

[0111] The incremental time stamp unit is used to determine whether there is an incremental time stamp in the data table structure of the current version of the extraction database.

[0112] The index field unit is used to establish a unique index field if the data table structure of the current version of the extraction database does not have an incremental timestamp, and to determine the extracted data content through the unique index field.

[0113] Embodiment seven:

[0114] Figure 7 Schematic diagram of a terminal device provided by Embodiment 7 of the present invention. Figure 7 As shown, the terminal device 7 of this embodiment includes: a processor 70, a memory 71, and a computer program 72 stored in the memory 71 and executable on the processor 70, such as a program. When the processor 70 executes the computer program 72, the steps in the above-mentioned data extraction method embodiments are implemented, such as Figure 1 Alternatively, when the processor 70 executes the computer program 72, the functions of each module / unit in the above system embodiment are realized, for example Figure 4 The functions of modules 101 to 104 are shown.

[0115] Exemplarily, the computer program 72 may be divided into one or more modules / units, which are stored in the memory 71 and executed by the processor 70 to complete the present invention. The one or more modules / units may be a series of computer program instruction segments capable of completing specific functions, which are used to describe the execution process of the computer program 72 in the terminal device 7. For example, the computer program 72 may be divided into a judgment module, a generation module, a modification module, and an extraction module, and the specific functions of each module are as follows:

[0116] A judgment module, used to compare the data source table structure of the current version of the extraction database with the data source table structure of the previous version of the extraction database, and judge whether the data source table structure of the extraction database has changed;

[0117] A generating module, for generating a script file for modifying the target table structure according to the changed content if the data source table structure of the extraction database changes;

[0118] A modification module, used for modifying the target table structure according to the script file for modifying the target table structure, and generating a number extraction script file according to the modified target table structure;

[0119] An extraction module is used to extract data from the extraction database according to the extraction script file.

[0120] The terminal device 7 may be a computing device such as a desktop computer, a notebook, a PDA, or a cloud management server. The terminal device may include, but is not limited to, a processor 70 and a memory 71. Those skilled in the art will appreciate that Figure 7 It is only an example of the terminal device 7 and does not constitute a limitation on the terminal device 7. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the terminal device may also include input and output devices, network access devices, buses, etc.

[0121] The processor 70 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0122] The memory 71 may be an internal storage unit of the terminal device 7, such as a hard disk or memory of the terminal device 7. The memory 71 may also be an external storage device of the terminal device 7, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the terminal device 7. Further, the memory 71 may also include both an internal storage unit and an external storage device of the terminal device 7. The memory 71 is used to store the computer program and other programs and data required by the terminal device. The memory 71 may also be used to temporarily store data that has been output or is to be output.

[0123] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In actual applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the system can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned wireless terminal can refer to the corresponding process in the above-mentioned method embodiment, and will not be repeated here.

[0124] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0125] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0126] In the embodiments provided by the present invention, it should be understood that the disclosed system / terminal device and method can be implemented in other ways. For example, the system / terminal device embodiments described above are only schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the system or unit can be electrical, mechanical or other forms.

[0127] The units described as separate components may or may not be physically separated, and the components described as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0128] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0129] If the integrated module / unit is implemented in the form of a software functional unit and is set to be sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or system that can carry the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0130] The embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the protection scope of the present invention.

Claims

1. A data extraction method, characterized in that: include: Compare the data source table structure of the current version of the extraction database with the data source table structure of the previous version of the extraction database to determine whether the data source table structure of the extraction database has changed; If the data source table structure of the extraction database changes, a script file for modifying the target table structure is generated according to the changed content. The data extraction process is to extract the data in the extraction database and save it in the target database; Modify the target table structure according to the script file for modifying the target table structure, and generate a number extraction script file according to the modified target table structure; Extracting data from the extraction database according to the extraction script file; Wherein, extracting data from the extraction database according to the extraction script file comprises: Establishing a number extraction task, wherein the number extraction task includes an execution mode of the number extraction task, and the execution mode includes full extraction and incremental extraction; If the execution mode is incremental extraction, the extracted data content is determined according to the incremental timestamp of the metadata of the extraction database, and the extraction script file is run according to the extracted data content; Furthermore, if the execution mode is incremental extraction, before determining the extracted data content according to the incremental timestamp of the metadata of the extraction database, it also includes: Determine whether there is an incremental timestamp in the data table structure of the current version of the extraction database; If the data table structure of the current version of the extraction database does not have an incremental timestamp, a unique index field is established, and the extracted data content is determined by the unique index field.

2. The method according to claim 1, characterized in that: The comparing the data source table structure of the current version of the extraction database with the data source table structure of the previous version of the extraction database to determine whether the data source table structure of the extraction database has changed includes: Query the metadata of the extraction database through structured query language, and obtain the data source table structure of the current version according to the metadata; Obtain the data source table structure of the previous version of the sampling database stored in the XML file; Determine whether there are changed fields between the data source table structure of the current version and the data source table structure of the previous version, wherein the changed fields include added fields, deleted fields, and modified fields; If there are changed fields between the data source table structure of the current version and the data source table structure of the previous version, the data source table structure of the extraction database is changed.

3. The method according to claim 1, characterized in that The extracting data from the extraction database according to the extraction script file also includes: If the execution mode is full extraction, the extraction script file is run to extract all the data in the extraction database.

4. The method according to claim 1, characterized in that: Before comparing the data source table structure of the current version of the extraction database with the data source table structure of the previous version of the extraction database to determine whether the data source table structure of the extraction database has changed, the method further includes: Identify whether there is an XML file that saves the previous version of the data source table structure; If the XML file does not exist, connect to the database through the database connection interface, and search and obtain the corresponding data source table structure through the database; An XML file is created, and the data source table structure is saved in the XML file.

5. A data extraction system, characterized in that: include: A judgment module, used to compare the data source table structure of the current version of the extraction database with the data source table structure of the previous version of the extraction database, and judge whether the data source table structure of the extraction database has changed; A generating module, for generating a script file for modifying the target table structure according to the changed content if the data source table structure of the extraction database changes, wherein the data extraction process is to extract the data from the extraction database and save it in the target database; A modification module, used for modifying the target table structure according to the script file for modifying the target table structure, and generating a number extraction script file according to the modified target table structure; An extraction module, used for extracting data from the extraction database according to the extraction script file; Wherein, the extraction module comprises: A task establishment unit, used to establish a number extraction task, wherein the number extraction task includes an execution mode of the number extraction task, and the execution mode includes full extraction and incremental extraction; An incremental extraction unit, configured to determine the extracted data content according to the incremental timestamp of the metadata of the extraction database if the execution mode is incremental extraction, and to run the extraction script file according to the extracted data content; An incremental timestamp unit, used to determine whether there is an incremental timestamp in the data table structure of the current version of the extraction database; The index field unit is used to establish a unique index field if the data table structure of the current version of the extraction database does not have an incremental timestamp, and to determine the extracted data content through the unique index field.

6. The data extraction system according to claim 5, characterized in that: The judging module comprises: A structure query unit, used to query the metadata of the extraction database through structured query language, and obtain the data source table structure of the current version according to the metadata; An acquisition unit, used for acquiring a data source table structure of a previous version of the sampling database stored in an XML file; A judgment unit is used to judge whether there are changed fields between the data source table structure of the current version and the data source table structure of the previous version, and the changed fields include added fields, deleted fields and modified fields; if there are changed fields between the data source table structure of the current version and the data source table structure of the previous version, the data source table structure of the sampling database changes.

7. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 4 are implemented.

8. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 4 are implemented.

Images