Big data extraction method and device, computer device and storage medium

Abstract

The application relates to a big data extraction method and device, computer equipment, a storage medium and a computer program product. The method comprises the following steps: acquiring a system category and table information of a target business system; determining a database type of the target business system according to a first corresponding relationship between preset system categories and database types and the system category of the target business system; determining a target database field type of the target business system according to the database type of the target business system; extracting through a data dictionary extraction script to generate a data dictionary of the target business system according to the target database field type of the target business system and the table information of the target business system; and reading data in the target business system through a preset data collection script and the data dictionary of the target business system. The method can improve data extraction efficiency.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data storage, in particular to a big data extraction method and device, computer equipment, storage medium and computer program product. BACKGROUND

[0002] With the development of data storage technology, in order to realize the transfer storage of massive data, for example, storing the business data of each business system to a big data platform, big data extraction technology appears.

[0003] In the traditional technology, when the big data platform accesses the target business system, the database type of the target business system is determined from the operation and maintenance manufacturer during demand research, and the corresponding data collection script is written according to the database type of the target business system. When the big data platform executes the corresponding data collection script, the corresponding data is extracted from the target business system database.

[0004] However, in the traditional method, when the big data platform accesses each target business system with different database types, the data collection script corresponding to each database type needs to be written respectively, so that the big data platform can extract the data in the database of each target business system with different types, thereby reducing the data extraction efficiency to a certain extent. SUMMARY

[0005] Therefore, it is necessary to provide a big data extraction method and device, computer equipment, computer readable storage medium and computer program product capable of improving the data extraction efficiency.

[0006] In a first aspect, the present application provides a big data extraction method. The method comprises:

[0007] obtaining the system category and table information of the target business system;

[0008] determining the database type of the target business system according to a first correspondence relationship between the preset system category and the database type, and the system category of the target business system;

[0009] determining the target database field type of the target business system according to the database type of the target business system;

[0010] generating the data dictionary of the target business system by extracting through the data dictionary extraction script according to the target database field type of the target business system and the table information of the target business system;

[0011] reading the data in the target business system through the preset data collection script and the data dictionary of the target business system.

[0012] In one of the embodiments, the target database field type of the target business system is determined according to the database type of the target business system, comprising:

[0013] The target database field type of the target business system is determined according to the mapping relationship between the preset database field type and the target database field type, and the field type of the target business system.

[0014] In one of the embodiments, the data in the target business system is read through the preset data collection script and the data dictionary of the target business system, comprising:

[0015] The connection account password and the connection address of the target business system are obtained;

[0016] The connection account password of the target business system is decrypted to obtain the decrypted connection account password;

[0017] The drive configuration of the target business system is determined according to the second corresponding relationship between the preset database type and the drive configuration, and the database type of the target business system.

[0018] The data collection script is driven to execute through the drive configuration of the target business system according to the data dictionary of the target business system and the decrypted connection account password, and the data in the target business system is read by connecting to the connection address.

[0019] In one of the embodiments, the connection account password of the target business system is obtained, comprising:

[0020] The connection account password of the target business system is determined according to the system category of the target business system and the third corresponding relationship between the preset system category and the connection account password.

[0021] In one of the embodiments, the connection account password of the target business system is decrypted to obtain the decrypted connection account password, comprising:

[0022] The ciphertext obtained by encrypting the connection account password of the target business system through the key is obtained;

[0023] The ciphertext is decrypted through the key to obtain the decrypted connection account password.

[0024] In one of the embodiments, the method further comprises:

[0025] The field type of the business system of each different database type is obtained;

[0026] According to the field type of the table in each business system, a field type mapped to the target database is generated, and a mapping relationship between the preset database field type and the target database field type is obtained.

[0027] In a second aspect, the present application further provides a big data extraction device. The device comprises:

[0028] a business system information acquisition module, configured to acquire the system category and table information of a target business system;

[0029] a database type acquisition module, configured to determine the database type of the target business system according to a first corresponding relationship between the preset system category and database type and the system category of the target business system;

[0030] a field type determination module, configured to determine the target database field type of the target business system according to the database type of the target business system;

[0031] a data dictionary determination module, configured to generate the data dictionary of the target business system by data dictionary extraction scripts according to the target database field type of the target business system and the table information of the target business system;

[0032] a data reading module, configured to read the data in the target business system by the preset data collection scripts and the data dictionary of the target business system.

[0033] In a third aspect, the present application further provides a computer device. The computer device comprises a memory and a processor, the memory stores a computer program, and the processor realizes the following steps when executing the computer program:

[0034] acquiring the system category and table information of a target business system;

[0035] determining the database type of the target business system according to a first corresponding relationship between the preset system category and database type and the system category of the target business system;

[0036] determining the target database field type of the target business system according to the database type of the target business system;

[0037] generating the data dictionary of the target business system by data dictionary extraction scripts according to the target database field type of the target business system and the table information of the target business system;

[0038] reading the data in the target business system by the preset data collection scripts and the data dictionary of the target business system.

[0039] In a fourth aspect, the present application provides a computer readable storage medium. The computer readable storage medium has a computer program stored thereon, and the computer program, when executed by a processor, implements the following steps:

[0040] obtaining system category and table information of a target business system;

[0041] determining database type of the target business system according to a first correspondence relationship between system category and database type and the system category of the target business system;

[0042] determining target database field type of the target business system according to the database type of the target business system;

[0043] generating a data dictionary of the target business system by data dictionary extraction script according to the target database field type of the target business system and the table information of the target business system;

[0044] reading data in the target business system by a preset data collection script and the data dictionary of the target business system.

[0045] In a fifth aspect, the present application provides a computer program product. The computer program product comprises a computer program, and the computer program, when executed by a processor, implements the following steps:

[0046] obtaining system category and table information of a target business system;

[0047] determining database type of the target business system according to a first correspondence relationship between system category and database type and the system category of the target business system;

[0048] determining target database field type of the target business system according to the database type of the target business system;

[0049] generating a data dictionary of the target business system by data dictionary extraction script according to the target database field type of the target business system and the table information of the target business system;

[0050] reading data in the target business system by a preset data collection script and the data dictionary of the target business system.

[0051] The big data extraction method, device, computer device, storage medium and computer program product can obtain the system category and table information of the target business system, determine the database type of the target business system according to the system category of the target business system and the first correspondence relationship, and determine the target database field type of the target business system according to the database type of the target business system, so as to extract through the data dictionary script according to the target database field type of the target business system and the table information of the target business system, and further generate the data dictionary of the target business system. According to the data dictionary of the target business system and the preset data collection script, the data in the target business system can be read, and the data extraction of the big data platform to the target business system is completed. For any target business system, even if the database types are different, the data collection scripts corresponding to the database types do not need to be written respectively, and the big data platform can extract the data in the databases of different types of target business systems, so that the efficiency of extracting data from the business system by the big data platform is improved to a certain extent. BRIEF DESCRIPTION OF DRAWINGS

[0052] Figure 1 An application environment diagram of the big data extraction method in an embodiment;

[0053] Figure 2 A flowchart of the big data extraction method in an embodiment;

[0054] Figure 3 A flowchart of the big data extraction method in another embodiment;

[0055] Figure 4 A structural block diagram of the big data extraction device in an embodiment;

[0056] Figure 5 An internal structure diagram of the computer device in an embodiment. DETAILED DESCRIPTION

[0057] In order to make the purpose, technical scheme and advantages of the present application clearer, the present application is further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.

[0058] The big data extraction method provided in the embodiments of the present application can be applied to, for example, Figure 1The application environment shown. Among them, including big data platform 10 and several business systems 20, big data platform 10 is used as enterprise server, big data platform 10 can be Hadoop distributed file system, each business system 20 is a database storing different business scenario data, including data under specific business scenario. For example, sales business system 20, the generated is a large number of sales data. Target business system 20 is the business system 20 that big data platform 10 needs to extract data when conducting data analysis. When responding to the data analysis instruction of the user operation, then big data platform 10 determines the target business system 20 according to the data analysis instruction, and then determines the corresponding database type and target database field type according to the system category of the target business system 20. According to the target database field type and the table information of the target business system 20, the data dictionary extraction script is extracted, so as to generate the data dictionary of the target business system 20. Finally, through the preset data collection script and the data dictionary of the target business system 20, the data in the target business system 20 is read.

[0059] In one embodiment, as shown in Figure 2 A big data extraction method is provided, and the method is applied to Figure 1 Big data platform in example, including the following steps:

[0060] Step 202, obtaining the system category and table information of the target business system.

[0061] The business system is a database for storing relevant business operations, and the data generated by the business system is different according to different business scenarios. The system category of the business system is the business scenario category of the business system, and the business scenarios are divided according to the specific stored business data. The same business scenario can have multiple business systems, and the system categories of the corresponding business systems are the same, and the database structures of the business systems of each same system category are consistent. The target business system is a business system that needs to extract data for business analysis by the big data platform, and the target business system includes a business system newly connected to the big data platform and a business system with updated data. Each business system has its own data source, which is generated by a large number of users in different business scenarios. Taking a power enterprise as an example, the business system includes a power sales business system and a power distribution business system, and the power sales business system includes residential power sales data. The residential power sales data is sales data generated by residents when purchasing power. The power distribution business system includes the resident's meter number and the meter installation address. When the power enterprise has a business analysis requirement, the big data platform responds to the business request by inputting the business request to the big data platform, determining the target business system from each business system, and the system category and table information of the target business system. In one embodiment, the user input business request is obtained, the business request includes the keywords of A area and residential power consumption, the big data platform responds to the business request to determine that the target business system includes the power distribution business system and the power sales business system, and the system category and table information of the power distribution business system and the power sales business system are obtained.

[0062] In one big data platform, the business system transmitting data thereto can be compiled by multiple database languages, and each database language can be Mysql, Oracle, SqlServer, etc.

[0063] For various database languages, data is stored in the form of a table. The table information of the business system refers to the basic information of the table corresponding to the database language used by the current business system. Optionally, the table information includes a table name. In addition, each business system usually includes a large number of table structures, and usually, the table name of the target business system needs to be determined before the big data platform extracts the data of the business system.

[0064] Step 204, according to the first correspondence relationship between the preset system category and the database type, and the system category of the target business system, the database type of the target business system is determined.

[0065] The system category and the database type have a first correspondence relationship, and the big data platform has pre-stored the first correspondence relationship before extracting data. When extracting data of the target business system, the database type of the target business system is determined according to the first correspondence relationship and the system category of the target business system. The first correspondence relationship is pre-stored in the big data platform, and when extracting data in the future, the big data platform can determine the database type of the target business system when the system category of the target business system is determined. The database type of the business system refers to the database language category used by the current business system.

[0066] In step 206, the target database field type of the target business system is determined according to the database type of the target business system.

[0067] The various types of databases have corresponding field types, and the field type of the business system is the field type corresponding to the database used by the current business system. An external table is created on the big data platform, and the field type of the external table is referred to as the target database field type. In order to store the data in the target business system on the big data platform, it is necessary to convert the field type of the target business system to the target database field type. For example, when the big data platform creates an external table in hive, the field type of the target business system needs to be converted to the hive database field type when extracting.

[0068] The field type is a data type in a database, including binary data type, character string type and Unicode data type, wherein the binary data type is, for example, Binary, Varbinary, Image, etc., the character data type includes Char, Varchar and Text, and the Unicode data type includes Nchar, Nvarchar and Ntext, etc.

[0069] In one embodiment, the target database field type of the target business system is determined according to the database type of the target business system. That is, the field type of the target business system is determined according to the database type of the target business system, and the target database field type is determined according to the field type of the target business system.

[0070] In step 208, the data dictionary of the target business system is extracted by a data dictionary extraction script according to the target database field type of the target business system and the table information of the target business system, and the data dictionary of the target business system is generated.

[0071] The data dictionary extraction script, i.e., a data dictionary generation script, can generate a data dictionary of the target business system according to a target database field type of the target business system and table information of the target business system. The data dictionary of the target business system refers to a definition and description of data items, data structures, data flow, data storage, and processing logic of data in the target business system. The data dictionary of the target business system includes description and definition of data items, data structures, data storage, and processing logic in the target business system. Optionally, the data structure includes the target database field type.

[0072] In one embodiment, the table information includes an ownership mode and a table name. The ownership mode and the table name are extracted by the data dictionary extraction script, and the target database field type of the target business system, so as to generate the data dictionary of the target business system. The data dictionary of the target business system includes the ownership mode, the table name, and the target database field type.

[0073] The ownership mode and the table name are basic information of the target business system, and need to be obtained by the big data platform from the data dictionary in real time according to the target business system each time. The mapping relationship between the preset database field type and the target database field type is stored on the big data platform, and the big data platform does not need to obtain the target business system in real time each time, but queries the basic information of the target business system each time when data needs to be extracted, so as to obtain the data dictionary of the target business system.

[0074] In step 210, data in the target business system is read by a preset data collection script and the data dictionary of the target business system.

[0075] The data collection script is an extraction script for extracting data in each database, and includes a sqoop data extraction script. The data collection script is preset on the big data platform. When the big data platform needs to extract data in the target business system database, the data collection script extracts data in the target business system database through data information displayed by the data dictionary of the target business system. When the data is extracted, the big data platform executes the data collection script. The data collection script automatically extracts corresponding data according to the input data dictionary of the target business system in the execution process.

[0076] In the above data extraction method, the big data platform acquires the system category and table information of the target business system, determines the database type of the target business system according to the system category of the target business system and the first correspondence relationship, and determines the target database field type of the target business system according to the database type of the target business system, so as to extract through a data dictionary script according to the target database field type of the target business system and the table information of the target business system, and further generate a data dictionary of the target business system. According to the data dictionary of the target business system and the preset data collection script, the data in the target business system can be read, and the data extraction of the big data platform to the target business system is completed. For any target business system, even if the database types are different, data collection scripts corresponding to each database type do not need to be written respectively, and the big data platform can extract data in databases of different types of target business systems, thereby improving the efficiency of data extraction of the big data platform from the business system to a certain extent.

[0077] In one embodiment, determining the target database field type of the target business system according to the database type of the target business system comprises: determining the target database field type of the target business system according to the mapping relationship between the preset database field type and the target database field type, and the field type of the target business system.

[0078] Wherein, the ownership mode is the mode of the table in the database, one database can have multiple ownership modes; one ownership mode can have 0 or multiple tables. For example, the power distribution business system includes the electric meter number of residents and the electric meter number of industry, which are both electric meter numbers, and the difference lies in that one ownership mode is for residents and the other ownership mode is for industry. The table name refers to the name of each table in the database, and the names of the tables should be kept different to facilitate subsequent table lookup according to the table name.

[0079] The big data platform pre-stores the mapping relationship between the database field type and the target database field type, and the mapping relationship refers to the corresponding conversion relationship between the database field type of the business system and the target database field type. When the big data platform determines the database type of the target business system, the target database field type corresponding to the field type of the target business system is found in the preset mapping relationship between the database field type and the target database field type. For example, when the big data platform determines that the database field type in the target business system is integer type data, the database field type of the target business system is found to be byte type in the mapping relationship, and the data in the target business system with the database field type of integer type is converted into the corresponding byte type data of the target database.

[0080] In this embodiment, the target database field type is determined through a mapping relationship between a preset database field type and the target database field type and a field type of the target business system.

[0081] In one embodiment, the data in the target business system is read through a preset data collection script and a data dictionary of the target business system, including: obtaining a connection account password and a connection address of the target business system; decrypting the connection account password of the target business system to obtain a decrypted connection account password; determining a driver configuration of the target business system according to a second correspondence relationship between preset database types and driver configurations and a database type of the target business system; and driving the data collection script to execute through the driver configuration of the target business system according to the data dictionary of the target business system and the decrypted connection account password, connecting to the connection address to read the data in the target business system.

[0082] The connection account password is a prerequisite for obtaining a large amount of actual data in the business system. Each connection account password is usually in an encrypted state, and only the decrypted connection account password can normally obtain data in the business system. When the big data platform extracts data in the target business system, the connection account password and the connection address of the target business system are first obtained. Since the connection account password is in an encrypted state, it needs to be decrypted first to obtain the decrypted connection account password.

[0083] The big data platform also has a second correspondence relationship between database types and driver configurations. The second correspondence relationship includes each database type, the driver configuration corresponding to each database type, and the matching / corresponding relationship therebetween. The driver configuration is an engine for driving data extraction, and is used to drive the data collection script to extract data from the target business system. When the big data platform extracts the connection account password of the target business system, it also needs to find the driver configuration corresponding to the target business system in the second correspondence relationship according to the database type of the target business system.

[0084] The preset data collection script can determine the data architecture of the target business system according to the data dictionary of the target business system.

[0085] Then, the big data platform connects the data collection script to the target business system according to the decrypted connection account password, the data dictionary of the target business system, the connection address, and the corresponding driver configuration, and drives the data collection script to execute through the driver configuration. In the process of executing the data collection script, the data in the target business system is continuously read.

[0086] In this embodiment, the big data platform extracts data in the target business system by decrypting the obtained connection account password of the target business system, a connection address, and a driver configuration, driving a data collection script to extract data in the target business system, and extracting the data in the target business system to the big data platform.

[0087] In one embodiment, the connection account password of the target business system is obtained, including: determining the connection account password of the target business system according to a system category of the target business system and a third correspondence relationship between a preset system category and a connection account password.

[0088] In this embodiment, each business system has a corresponding connection account password, and each business system and the corresponding connection account password are stored in the third correspondence relationship on the big data platform. Before decryption, the connection account password stored in the third correspondence relationship is in an encrypted state. The big data platform finds the connection account password corresponding to the target business system from the third correspondence relationship before executing the data collection script.

[0089] In this embodiment, the connection account password corresponding to the target business system is obtained from the third correspondence relationship. At this time, the obtained connection account password is encrypted, so as to avoid configuring the password in plaintext on the data collection script during data extraction, which is beneficial to improve the security of data extraction.

[0090] In one embodiment, the connection account password of the target business system is decrypted to obtain a decrypted connection account password, including: obtaining ciphertext obtained by encrypting the connection account password by a key; and decrypting the ciphertext by the key to obtain the decrypted connection account password.

[0091] In this embodiment, the target business system is encrypted by using a symmetric encryption method, and the connection account password of the target business system is first encrypted by a key to obtain encrypted ciphertext. When the big data platform obtains the connection account password of the target business system in an encrypted state by the data collection script, the same key is used to decrypt the connection account password to obtain the decrypted connection account password. The data collection script can extract actual data in the target business system according to the decrypted connection account password.

[0092] In this embodiment, the ciphertext of the connection account password is decrypted by the key to obtain the decrypted connection account password, which is convenient for the big data platform to extract data in the target business system according to the data collection script.

[0093] In one embodiment, the big data extraction method comprises: obtaining field types in business systems of different database types; generating field types mapped to a target database according to the field types in each of the business systems, to obtain a mapping relationship between preset database field types and target database field types.

[0094] In the embodiment, before the mapping relationship between the preset database field types and the target database field types is obtained, the field types of tables in the business systems of different database types are obtained, and the field types of the tables in the target database are generated according to the field types of the tables in each of the business systems, to obtain target database field types of each of the business systems. The field types of the tables in the target database are generated by analyzing the field types of the tables in the business systems and the field types of the tables in the target database, determining that the field types of the tables in the business systems are stored in the data types to be converted in the target database, obtaining the mapping relationship between the preset database field types and the target database field types, and pre-storing the mapping relationship in the big data platform.

[0095] In the embodiment, the field types of the business systems of different database types are generated to the field types mapped to the target database, to obtain the mapping relationship between the preset database field types and the target database field types, so that the big data platform can subsequently generate a data dictionary of the target business system.

[0096] In one embodiment, as shown in FIG. 1, Figure 3 the big data extraction method comprises steps S10-S40, wherein:

[0097] S10: Obtain configuration basic information, the configuration basic information comprising a system category of a target business system and table information, the table information comprising an ownership mode and a table name.

[0098] S20: Dynamically generate a data dictionary of the target business system according to the solidified configuration content, the configuration basic information, and a data dictionary extraction script.

[0099] The solidified configuration content comprises a first corresponding relationship between preset system categories and database types, a mapping relationship between preset database field types and target database field types, a second corresponding relationship between preset database types and driver configurations, and a third corresponding relationship between preset system categories and connection account passwords. When the data dictionary of the target business system is generated, only the first corresponding relationship and the mapping relationship in the solidified configuration content are read.

[0100] S30: According to the system category of the target business system, the second correspondence relationship and the third correspondence relationship are searched to determine the corresponding driving configuration and connection account password of the target business system, the connection account password of the target business system is decrypted to obtain the decrypted connection account password; according to the decrypted connection account password and the data dictionary of the target business system, the data collection script is driven through the driving configuration of the target business system to connect to the connection address to read the data in the target business system.

[0101] S40: When the data collection script is executed, the data of the target business system is extracted to the big data platform.

[0102] When the data of the target business system is extracted to the big data platform, the data landing is realized, and the data extraction work is completed.

[0103] In the embodiment, the big data platform obtains the system category and table information of the target business system, determines the database type of the target business system according to the system category of the target business system and the first correspondence relationship, and determines the target database field type of the target business system according to the database type of the target business system, so as to extract through the data dictionary script according to the target database field type of the target business system and the table information of the target business system, and further generate the data dictionary of the target business system. According to the data dictionary of the target business system and the preset data collection script, the data in the target business system can be read, and the data extraction of the big data platform to the target business system is completed. For any target business system, even if the database types are different, it is not necessary to write data collection scripts corresponding to each database type respectively, so that the big data platform can extract data in databases of different types of target business systems, thereby improving the efficiency of data extraction of the big data platform from the business system to a certain extent.

[0104] It should be understood that although each step in the flowchart involved in each embodiment as described above is displayed in sequence according to the arrow, these steps are not necessarily executed in sequence according to the arrow. Unless otherwise stated herein, the execution of these steps has no strict sequence limitation, and these steps can be executed in other orders. Moreover, at least part of the steps in the flowchart involved in each embodiment as described above can include multiple steps or stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily sequential, but can be alternately or alternately executed with at least part of other steps or steps or stages in other steps.

[0105] Based on the same inventive concept, the embodiments of the present application also provide a big data extraction device for implementing the above-mentioned big data extraction method. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme described in the above-mentioned method, so the specific limitations in one or more big data extraction device embodiments provided below can refer to the limitations of the big data extraction method in the foregoing, which will not be repeated here.

[0106] In one embodiment, as shown in Figure 4 A big data extraction device is provided, comprising: a business system information acquisition module 301, a database type acquisition module 302, a field type determination module 303, a data dictionary determination module 304, and a data reading module 305, wherein:

[0107] The business system information acquisition module 301 is configured to acquire the system category and table information of the target business system.

[0108] The database type acquisition module 302 is configured to determine the database type of the target business system according to a first correspondence relationship between the preset system category and the database type, and the system category of the target business system.

[0109] The field type determination module 303 is configured to determine the target database field type of the target business system according to the database type of the target business system.

[0110] The data dictionary determination module 304 is configured to generate the data dictionary of the target business system by extracting through a data dictionary extraction script according to the target database field type of the target business system and the table information of the target business system.

[0111] The data reading module 305 is configured to read the data in the target business system through a preset data collection script and the data dictionary of the target business system.

[0112] In one embodiment, the field type determination module is further configured to determine the target database field type of the target business system according to a mapping relationship between the preset database field type and the target database field type, and the field type of the target business system.

[0113] In one embodiment, the data reading module comprises an account address acquisition module, an account decryption module, a driver configuration determination module, and a data connection module, wherein:

[0114] The account address acquisition module is configured to acquire the connection account password and the connection address of the target business system.

[0115] An account decryption module is configured to decrypt the connection account password of the target business system to obtain a decrypted connection account password.

[0116] A driver configuration determination module is configured to determine the driver configuration of the target business system according to a second correspondence relationship between preset database types and driver configurations and the database type of the target business system.

[0117] A data connection module is configured to drive the data collection script to execute through the driver configuration of the target business system according to the data dictionary of the target business system and the decrypted connection account password, and connect to the connection address to read data in the target business system.

[0118] In one embodiment, the account address acquisition module is further configured to determine the connection account password of the target business system according to a third correspondence relationship between preset system categories and connection account passwords and the system category of the target business system.

[0119] In one embodiment, the account decryption module includes a ciphertext acquisition module and a decrypted account acquisition module, wherein:

[0120] The ciphertext acquisition module is configured to acquire ciphertext obtained by encrypting the connection account password by a key by the target business system.

[0121] The decrypted account acquisition module is configured to decrypt the ciphertext by the key to obtain the decrypted connection account password.

[0122] In one embodiment, the big data extraction apparatus further includes a field type acquisition module and a mapping relationship acquisition module, wherein:

[0123] The field type acquisition module is configured to acquire the field types of the business systems of different database types.

[0124] The mapping relationship acquisition module is configured to respectively generate field types mapped to a target database according to the field types of the business systems to obtain a mapping relationship between preset database field types and target database field types.

[0125] Each module in the above big data extraction apparatus can be realized by software, hardware, and combinations thereof, in whole or in part. Each module can be embedded in or independent of a processor in a computer device in a hardware form, or can be stored in a memory in a computer device in a software form, so as to be called and executed by a processor to perform operations corresponding to each module.

[0126] In one embodiment, a computer device is provided, which can be a server, and an internal structure diagram of the computer device can be as shown in Figure 5As shown. The computer device includes a processor, a memory and a network interface connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operating system and the computer program in the non-volatile storage medium to run. The network interface of the computer device is used to communicate with the external terminal through the network connection. The computer program is executed by the processor to implement a big data extraction method.

[0127] Those skilled in the art can understand that, Figure 5 The structure shown in the figure is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device can include more or fewer components than those shown in the figure, or combine certain components, or have a different component arrangement.

[0128] In one embodiment, a computer device is also provided, including a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the steps in each of the method embodiments.

[0129] In one embodiment, a computer readable storage medium is provided, which stores a computer program, and the computer program is executed by a processor to implement the steps in each of the method embodiments.

[0130] In one embodiment, a computer program product is provided, including a computer program, and the computer program is executed by a processor to implement the steps in each of the method embodiments.

[0131] It should be noted that the user information (including but not limited to user equipment information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or authorized by all parties.

[0132] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when the computer program is executed, the processes of the above-mentioned embodiments of the methods can be included. Any reference to memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (Read-Only Memory, ROM), magnetic tape, floppy disk, flash memory, optical storage, high-density embedded non-volatile memory, resistive memory (ReRAM), magnetoresistive random access memory (Magnetoresistive Random Access Memory, MRAM), ferroelectric memory (Ferroelectric Random Access Memory, FRAM), phase change memory (Phase Change Memory, PCM), graphene memory, etc. Volatile memory can include random access memory (Random Access Memory, RAM) or external cache memory, etc. As an illustration but not limitation, RAM can be in various forms, such as static random access memory (Static Random Access Memory, SRAM) or dynamic random access memory (Dynamic Random Access Memory, DRAM), etc. The database involved in the embodiments provided in the present application can include at least one of a relational database and a non-relational database. The non-relational database can include a distributed database based on a block chain, etc., without being limited thereto. The processor involved in the embodiments provided in the present application can be a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., without being limited thereto.

[0133] Any combination of the technical features of the above embodiments can be made. In order to make the description simple, all possible combinations of the technical features in the above embodiments are not described, however, as long as the combination of the technical features does not exist contradictory, it should be considered as the scope of the present application.

[0134] The above embodiments only express several implementation manners of the present application, and the description is more specific and detailed, but it should not be understood as a limitation on the scope of the patent of the present application. It should be pointed out that for ordinary skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are within the scope of protection of the present application. Therefore, the protection scope of the present application should be subject to the appended claims.

Claims

1. A big data extraction method, characterized in that, The method comprises: obtaining system category and table information of a target business system, the table information comprising ownership mode and table name; determining database type of the target business system according to a preset first correspondence relationship between system category and database type; determining field type of the target business system according to database type of the target business system, and determining target database field type of the target business system according to a preset mapping relationship between database field type and target database field type, and the field type of the target business system; extracting ownership mode, table name and target database field type through a data dictionary extraction script according to target database field type of the target business system and table information of the target business system, to generate a data dictionary of the target business system; obtaining connection address of the target business system; determining connection account password of the target business system according to system category of the target business system and a preset third correspondence relationship between system category and connection account password; decrypting the connection account password of the target business system to obtain decrypted connection account password; determining driver configuration of the target business system according to a preset second correspondence relationship between database type and driver configuration, and database type of the target business system; driving a data collection script to execute through the driver configuration of the target business system according to the data dictionary of the target business system and the decrypted connection account password, to connect to the connection address to read data in a database of the target business system.

2. The method of claim 1, wherein, decrypting the connection account password of the target business system to obtain decrypted connection account password comprises: obtaining ciphertext obtained by encrypting the connection account password through a key by the target business system; decrypting the ciphertext through the key to obtain the decrypted connection account password.

3. The method of claim 1, wherein, The method further comprises: obtaining field type of business system of each different database type; generating field type mapped to a target database according to field type of a table in each business system, to obtain a mapping relationship between preset database field type and target database field type.

4. The method according to any one of claims 1 to 3, characterized in that, The data dictionary of the target business system comprises description and definition content of data item, data structure, data storage and processing logic in the target business system.

5. A big data extraction apparatus characterized by comprising: The apparatus comprises: a business system information obtaining module configured to obtain system category and table information of a target business system, the table information comprising ownership mode and table name; a database type obtaining module configured to determine database type of the target business system according to a preset first correspondence relationship between system category and database type, and system category of the target business system; a field type determining module configured to determine field type of the target business system according to database type of the target business system, and determine target database field type of the target business system according to a preset mapping relationship between database field type and target database field type, and the field type of the target business system; The data dictionary determination module is configured to extract a home schema, a table name, and a target database field type through a data dictionary extraction script according to a target database field type of the target business system and table information of the target business system, and generate a data dictionary of the target business system. The data reading module is configured to obtain a connection address of the target business system; determine a connection account password of the target business system according to a system category of the target business system and a third correspondence relationship between a preset system category and a connection account password; decrypt the connection account password of the target business system to obtain a decrypted connection account password; determine a drive configuration of the target business system according to a second correspondence relationship between a preset database type and a drive configuration and a database type of the target business system; and drive a data collection script to execute through the drive configuration of the target business system according to the data dictionary of the target business system and the decrypted connection account password, connect to the connection address, and read data in a database of the target business system.

6. The apparatus of claim 5, wherein, The data reading module includes an account decryption module configured to obtain ciphertext obtained by encrypting the connection account password through a key by the target business system; and decrypt the ciphertext through the key to obtain the decrypted connection account password.

7. The apparatus of claim 5, wherein, The big data extraction device further includes a field type acquisition module and a mapping relationship acquisition module, wherein: The field type acquisition module is configured to acquire field types of business systems of different database types. The mapping relationship acquisition module is configured to generate field types mapped to a target database according to field types of tables in each of the business systems, and obtain a mapping relationship between preset database field types and target database field types.

8. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that, The processor executes the computer program to implement the steps of the method of any one of claims 1 to 4.

9. A computer readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method of any one of claims 1 to 4.

10. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method of any one of claims 1 to 4.

Images