A big data circulation method, device, electronic device and storage medium

Abstract

The present invention provides a big data flow method, device, electronic device and storage medium. If the data flow direction of the data flow task corresponding to the preset scheduling time is data extraction, the corresponding data is extracted from the source database indicated by the data extraction configuration information to the corresponding first target data table in the Hadoop database indicated by the data extraction configuration information; the first target data table is reorganized according to the data extraction conversion configuration rule and the data extraction configuration information; if it is data push, the corresponding second target data table in the Hadoop database indicated by the data push configuration information is reorganized according to the data push configuration information and the data push conversion configuration rule matching the data flow task, and then its data is pushed to the corresponding source data table in the corresponding source database. The present invention can realize the rapid and efficient flow of data between Hadoop and a relational database.

Description

Technical Field

[0001] The present invention relates to the field of big data technology, and more specifically, to a big data circulation method, device, electronic device and storage medium. Background Art

[0002] With the advent of the big data era, modern information technology represented by the Internet is fundamentally changing the traditional banking business model. The characteristics of massive, diversified, fast-transmission and value-added data have brought new challenges to the market competition of commercial banks. It is imperative to build a diversified ecosystem with cloud computing and big data as the core. Hadoop provides big data solutions for commercial banks with its low software and hardware costs and powerful parallel computing capabilities. Although Hadoop has many advantages in storing data, it is still difficult to achieve fast and efficient data flow between Hadoop and relational databases.

[0003] Therefore, how to provide a data flow method to achieve fast and efficient data flow between Hadoop and relational databases is an urgent problem to be solved in this application. Summary of the invention

[0004] In view of this, the present invention provides a big data circulation method, device, electronic device and storage medium, so as to realize fast and efficient circulation of data between Hadoop and relational database.

[0005] The first aspect of the present invention discloses a big data circulation method, the method comprising:

[0006] When the preset scheduling time is reached, obtaining a data flow task corresponding to the preset scheduling time, wherein the data flow task carries a data flow task number;

[0007] Determine whether the data flow direction of the data flow task is data extraction or data push;

[0008] If the data flow direction of the data flow task is data extraction, obtaining data extraction configuration information matching the data flow task number;

[0009] Extracting corresponding data from the source database indicated by the data extraction configuration information to a corresponding first target data table in the Hadoop database indicated by the data extraction configuration information;

[0010] Acquire a data extraction conversion configuration rule that matches the data extraction configuration information, and reorganize data in the first target data table according to the data extraction conversion configuration rule and the data extraction configuration information;

[0011] If the data flow direction of the data flow task is data push, obtain data push configuration information matching the data flow task number, and obtain data push conversion configuration rules matching the data push configuration information;

[0012] According to the data push conversion configuration rule and the data push configuration information, data is reorganized for the corresponding second target data table in the Hadoop database indicated by the data push configuration information, and its corresponding data is pushed to the corresponding source data table in the source database indicated by the data push information.

[0013] Optionally, the data extraction configuration information includes database extraction configuration information and data table extraction configuration information, and extracting corresponding data from a source database indicated by the data extraction configuration information to a corresponding first target data table in a Hadoop database indicated by the data extraction configuration information includes:

[0014] Extract the data in the source data table indicated by the data table extraction configuration information from the source database indicated by the database extraction configuration information to the first target data table indicated by the data table extraction configuration information in the Hadoop database indicated by the database extraction configuration information.

[0015] Optionally, the acquiring a data extraction conversion configuration rule matching the data extraction configuration information; and reorganizing data in the first target data table according to the data extraction conversion configuration rule and the data extraction configuration information, comprises:

[0016] Acquire a data extraction conversion configuration rule that matches the data extraction configuration information, wherein the data extraction conversion configuration rule includes multiple data processing methods;

[0017] The first target data table is reorganized according to the various data processing modes in the data extraction conversion configuration rules and the data table extraction configuration information.

[0018] Optionally, the data push configuration information includes database push configuration information and data table push configuration information, and the step of reorganizing the data of a corresponding second target data table in the Hadoop database indicated by the data push configuration information according to the data push conversion configuration rule and the data push configuration information, and pushing the corresponding data to a corresponding source data table in the source database indicated by the data push information includes:

[0019] According to the various data processing modes indicated by the data push conversion configuration rule and the data table push configuration information, data reorganization is performed on the second target data table indicated by the data table push configuration information in the Hadoop database indicated by the database push configuration information;

[0020] The data in the second target data table after data reorganization is pushed to the source data table indicated by the data table push configuration information in the source database indicated by the database push configuration information.

[0021] A second aspect of the present invention discloses a big data circulation device, the device comprising:

[0022] A first acquisition unit is used to acquire a data flow task corresponding to the preset scheduling time when the preset scheduling time is reached, wherein the data flow task carries a data flow task number;

[0023] A judging unit, used to judge whether the data flow direction of the data flow task is data extraction or data push;

[0024] A second acquisition unit is used to acquire data extraction configuration information matching the data flow task number if the data flow direction of the data flow task is data extraction;

[0025] A data extraction unit, configured to extract corresponding data from a source database indicated by the data extraction configuration information to a corresponding first target data table in a Hadoop database indicated by the data extraction configuration information;

[0026] a first data reorganization unit, configured to obtain a data extraction conversion configuration rule matching the data extraction configuration information; and to reorganize data on the first target data table according to the data extraction conversion configuration rule and the data extraction configuration information;

[0027] A third acquisition unit is used to acquire data push configuration information matching the data flow task number and acquire data push conversion configuration rules matching the data push configuration information if the data flow direction of the data flow task is data push;

[0028] The second data reorganization unit is used to reorganize the data of the corresponding second target data table in the Hadoop database indicated by the data push configuration information according to the data push conversion configuration rule and the data push configuration information, and push the corresponding data to the corresponding source data table in the source database indicated by the data push information.

[0029] Optionally, the data extraction configuration information includes database extraction configuration information and data table extraction configuration information, and the data extraction unit includes:

[0030] The data extraction subunit is used to extract data in the source data table indicated by the data table extraction configuration information from the source database indicated by the database extraction configuration information to the first target data table indicated by the data table extraction configuration information in the Hadoop database indicated by the database extraction configuration information.

[0031] Optionally, the first data reassembly unit includes:

[0032] A fourth acquisition unit, configured to acquire a data extraction conversion configuration rule matching the data extraction configuration information, wherein the data extraction conversion configuration rule includes a plurality of data processing modes;

[0033] The third data reorganization unit is used to reorganize the data of the first target data table according to the various data processing modes in the data extraction and conversion configuration rules and the data table extraction configuration information.

[0034] Optionally, the data push configuration information includes database push configuration information and data table push configuration information, and the second data reorganization unit includes:

[0035] a fourth data reorganization unit, configured to reorganize data on a second target data table indicated by the data table push configuration information in the Hadoop database indicated by the database push configuration information according to the various data processing modes indicated by the data push conversion configuration rule and the data table push configuration information;

[0036] The data pushing unit is used to push the data in the second target data table after data reorganization to the source data table indicated by the data table pushing configuration information in the source database indicated by the database pushing configuration information.

[0037] The third aspect of the present invention discloses an electronic device, which includes a processor and a memory, wherein the memory is used to store program codes and data for big data circulation, and the processor is used to call the program instructions in the memory to execute the big data circulation method disclosed in the first aspect of the present invention.

[0038] A fourth aspect of the present invention discloses a storage medium, which includes a storage program, wherein when the program is running, the device where the storage medium is located is controlled to execute the big data circulation method disclosed in the first aspect of the present invention as mentioned above.

[0039] The present invention provides a big data flow method, device, electronic device and storage medium. When a preset scheduling time is reached, a data flow task corresponding to the preset scheduling time is obtained, wherein the data flow task carries a data flow task number; it is determined whether the data flow direction of the data flow task is data extraction or data push; if the data flow direction of the data flow task is data extraction, data extraction configuration information matching the data flow task number is obtained; from a source database indicated by the data extraction configuration information, data in a corresponding source data table is extracted to a corresponding first target data table in a Hadoop database indicated by the data extraction configuration information; and a data extraction conversion configuration matching the data extraction configuration information is obtained. rules, and reorganize the data of the first target data table according to the data extraction conversion configuration rules and the data extraction configuration information; if the data flow direction of the data flow task is data push, obtain the data push configuration information matching the data flow task number and obtain the data push conversion configuration rules matching the data push configuration information, and reorganize the data of the corresponding second target data table in the Hadoop database indicated by the data push configuration information according to the data push conversion configuration rules and the data push configuration information, and push the corresponding data to the corresponding source data table in the source database indicated by the data push information, thereby realizing fast and efficient flow of data between the Hadoop database and the relational database. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0041] Figure 1 A schematic diagram of a flow chart of a big data circulation method provided by an embodiment of the present invention;

[0042] Figure 2 A schematic diagram of the structure of a large data flow device provided by an embodiment of the present invention;

[0043] Figure 3 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0044] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0045] The term "including" and its variations used herein are open inclusions, i.e., "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the following description.

[0046] It should be noted that the concepts such as "first" and "second" mentioned in the disclosure of the present invention are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0047] It should be noted that the modifications of "one" and "plurality" mentioned in the disclosure of the present invention are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0048] Hadoop: It is a distributed system infrastructure developed by the Apache Foundation. Users can develop distributed programs without understanding the underlying details of the distribution, and make full use of the performance of the cluster for high-speed computing and storage.

[0049] Sqoop: is an open source tool that is mainly used to transfer data between Hadoop (Hive) and source databases (Oracle database, MySQL database, Postgresql database, etc.). It can import data from a relational database into Hadoop's HDFS, and can also import HDFS data into a relational database.

[0050] See also Figure 1 , shows a flow chart of a data flow method provided by an embodiment of the present invention, the data flow method specifically comprises the following steps:

[0051] S101: When a preset scheduling time is reached, a data flow task corresponding to the preset scheduling time is obtained, wherein the data flow task carries a data flow task number.

[0052] In the embodiment of the present application, multiple data transfer tasks can be pre-set, and a corresponding preset scheduling time and a corresponding number can be set for each data transfer task. The purpose of setting the data transfer task is to realize the data transfer between the source database (relational database) and the Hadoop database.

[0053] It should be noted that the data flow task can be a data extraction task or a data push task. When the data flow task is a data extraction task, the data flow direction of the data flow task is data extraction; when the data flow task is a data push task, the data flow direction of the data flow task is data push.

[0054] It should be noted that when a preset scheduling time corresponds to multiple data flow tasks, the corresponding scheduling order and the corresponding scheduling status can also be set for each data flow task corresponding to the preset scheduling time. Among them, the scheduling status can be waiting for execution, importing, import completed, import error, import redo error, and non-scheduling state.

[0055] It should also be noted that corresponding scheduling batches can also be set.

[0056] In the specific process of executing step S101, when the current time of the data flow system reaches the preset scheduling time, a data flow task corresponding to the preset scheduling time is obtained, and the data flow task carries a data flow task number.

[0057] S102: Determine whether the data flow direction of the data flow task is data extraction or data push; if the data flow direction of the data flow task is data extraction, execute step S103; if the data flow direction of the data flow task is data push, execute step S106.

[0058] In the process of specifically executing step S102, after obtaining the data flow task corresponding to the preset scheduling time, it is further determined whether the data flow task is a data extraction task. If so, it can be determined that the data flow direction of the data flow task is data extraction; if not, it can be determined that the data flow task is a data push task, and then it can be determined that the data flow direction of the data flow task is data push.

[0059] S103: Acquire data extraction configuration information matching the data flow task number.

[0060] In the process of specifically executing step S103, when it is determined that the data flow direction of the data flow task is data extraction, pre-set data extraction configuration information matching the data flow task number can be obtained, wherein the data extraction configuration information includes database extraction configuration information and data table extraction configuration information.

[0061] In an embodiment of the present application, corresponding data extraction configuration information may be pre-set for each data extraction task, including database extraction configuration information and data table extraction configuration information.

[0062] Specifically, the database extraction configuration information includes the database configuration number, source database type, source database name, source database IP address, source database port number, source database schema, Hadoop database schema, etc. The data table extraction configuration information includes: data extraction conversion configuration rule number, data table configuration number, database configuration number, source data table name, target data table name, incremental import field (optional), data table partition field (optional), data table multi-time point field (optional), etc.

[0063] The data table type can be master or detail.

[0064] S104: Extract corresponding data from the source database indicated by the data extraction configuration information to a corresponding first target data table in the Hadoop database indicated by the data extraction configuration information.

[0065] In the process of specifically executing step S104, after obtaining the data extraction configuration information matching the data flow task number, the Sqoop component can be called to extract the data in the source data table corresponding to the source data table name indicated by the data table extraction configuration information from the source database corresponding to the source database name indicated by the database extraction configuration information to the target data table corresponding to the target data table name indicated by the data table extraction configuration information in the Hadoop database corresponding to the Hadoop database schema indicated by the database extraction configuration information (for ease of distinction, the target data table corresponding to the target data table name indicated by the data table extraction configuration information is called the first target data table).

[0066] In the embodiment of the present application, in addition to determining the corresponding source database according to the source database name indicated in the database configuration information, the corresponding source database may also be determined according to the source database IP address indicated in the database configuration information.

[0067] S105: Acquire a data extraction conversion configuration rule that matches the data extraction configuration information, and reorganize data in the first target data table according to the data extraction conversion configuration rule and the data extraction configuration information.

[0068] In an embodiment of the present application, corresponding data extraction and conversion configuration rules can be set in advance for each target data table in the Hadoop database. Specifically, the data extraction and conversion configuration number, character transcoding method, target storage format, whether the data is deduplicated, data compression method, data table type (master file, detail), preset map task concurrency number, whether to merge small files, whether to import incrementally, whether the data table is partitioned, whether there are multiple time points and other data processing methods can be included in the data extraction and conversion configuration rules.

[0069] It should be noted that the target storage format can be compress, Parquet, Hfile, txt, etc., which can be set according to actual applications and is not limited in the embodiments of the present application.

[0070] It should be noted that the concurrent data of the map task can be set to 2, 4, 6, or 12. It can be set according to the actual application and is not limited in the embodiment of the present application.

[0071] In the process of specifically executing step S105, after obtaining the data extraction configuration information that matches the data flow task number, the data extraction conversion configuration number indicated in the data table extraction configuration information in the data extraction configuration information can be used to obtain the data extraction conversion configuration rule that matches the data extraction conversion configuration number from the pre-set multiple data extraction conversion configuration rules.

[0072] After the corresponding data extraction and conversion configuration rules are obtained, data is reorganized on the first target data table according to each data processing mode and data table extraction configuration information in the data extraction and conversion configuration rules.

[0073] In an embodiment of the present application, if the data processing method in the data extraction and conversion configuration rules includes character transcoding method, target storage format, data deduplication, data compression method, data table type (main file), incremental import, data table partitioning, and multiple time points, the data in the first target data table is partitioned according to the data table partition field configured in the data table extraction configuration information, the incremental data in the first target data table is obtained according to the incremental access field configured in the data table extraction configuration information, the first target data table is distinguished between the main file and the details according to the data table type (main file) configured in the data table extraction configuration information, and multiple time points are set for the first target data table according to the multi-time point field configured in the data table extraction configuration information, and finally, the characters in the second target data table are transcoded according to the character transcoding method indicated by the data extraction and conversion configuration rules, the first target data table after character transcoding is deduplicated, and the data is compressed according to the data compression method.

[0074] Furthermore, in the embodiment of the present application, since Hadoop is a distributed system (Hadoop Distributed File System, HDFS) infrastructure developed by the Apache Foundation, after the first target data table in the Hadoop database is reorganized accordingly, if the data processing method in the data extraction and conversion configuration rule also includes merging small files on HDFS, then the small files related to the first target data table after the data reorganization in the Hadoop database are merged.

[0075] Furthermore, in an embodiment of the present application, data cleaning can be performed on the data in the first target data table after data reorganization in the Hadoop database according to pre-set data cleaning rules to check the data consistency in the first target data table after data reorganization, discover and correct errors in the data table, and process invalid values ​​and missing values.

[0076] S106: Acquire data push configuration information that matches the data flow task number, and acquire data push conversion configuration rules that match the data push configuration information.

[0077] In the process of specifically executing step S106, when it is determined that the data flow direction of the data flow task is data push, pre-set data push configuration information matching the data flow task number can be obtained, wherein the data push configuration information includes database push configuration information and data table push configuration information.

[0078] In an embodiment of the present application, corresponding data push configuration information may be set for each data push task, including database push configuration information and data table push configuration information.

[0079] Specifically, the database push configuration information includes the database configuration number, Hadoop database type, Hadoop database name, Hadoop database IP address, Hadoop database port number, Hadoop database schema, source database schema, etc. The data table push configuration information includes: data push conversion configuration rule number, data table configuration number, database configuration number, source data table name, target data table name, incremental import field (optional), data table partition field (optional), data table multi-time point field (optional), etc.

[0080] In an embodiment of the present application, corresponding data push conversion configuration rules can be set in advance for each target data table in the Hadoop database. Specifically, the data push conversion configuration number, character transcoding method, target storage format, whether the data is deduplicated, data compression method, data table type (master file, detail), preset map task concurrency number, whether to merge small files, whether to import incrementally, whether the data table is partitioned, whether there are multiple time points and other data processing methods can be set in the data push conversion configuration rules.

[0081] It should be noted that the target storage format can be compress, Parquet, Hfile, txt, etc., which can be set according to actual applications and is not limited in the embodiments of the present application.

[0082] It should be noted that the concurrent data of the map task can be set to 2, 4, 6, or 12. It can be set according to the actual application and is not limited in the embodiment of the present application.

[0083] S107: According to the data push conversion configuration rule and the data push configuration information, reorganize the data of the corresponding second target data table in the Hadoop database indicated by the data push configuration information, and push the corresponding data to the corresponding source data table in the source database indicated by the data push information.

[0084] In the process of specifically executing step S107, after obtaining the data push configuration information that matches the data flow task number, the data push conversion configuration number indicated in the data table push configuration information in the data push configuration information can be used to obtain the data push conversion configuration rule that matches the data push conversion configuration number from the pre-set multiple data push conversion configuration rules.

[0085] After obtaining the corresponding data push conversion configuration rules, the corresponding second target data table in the Hadoop database indicated by the data push configuration information is reorganized according to the various data processing methods and data table push configuration information in the data push conversion configuration rules, and the Sqoop component is called to push the data in the second target data table after data reorganization to the source data table indicated by the data table push information in the source database of the database push configuration information.

[0086] In an embodiment of the present application, if the data processing method in the data push conversion configuration rule includes character transcoding method, target storage format, data deduplication, data compression method, data table type (main file), incremental import, data table partitioning, and multiple time points, the data in the second target data table is partitioned according to the data table partition field configured in the data table push configuration information, the incremental data in the second target data table is obtained according to the incremental access field configured in the data table push configuration information, the second target data table is distinguished between the main file and the details according to the data table type (main file) configured in the data table push configuration information, and multiple time points are set for the second target data table according to the multi-time point field configured in the data table push configuration information, and finally, the characters in the second target data table are transcoded according to the character transcoding method indicated by the data push conversion configuration rule, the second target data table after character transcoding is deduplicated, and the data is compressed according to the data compression method.

[0087] Furthermore, in the embodiment of the present application, since Hadoop is a distributed system (Hadoop Distributed File System, HDFS) infrastructure developed by the Apache Foundation, if the data processing method in the data push conversion configuration rule also includes merging small files on HDFS, then the small files related to the second target data table after data reorganization in the Hadoop database are merged.

[0088] Furthermore, in an embodiment of the present application, data cleaning can be performed on the data in the second target data table after data reorganization in the Hadoop database according to pre-set data cleaning rules to check the data consistency in the second target data table after data reorganization, discover and correct errors in the data table, and process invalid values ​​and missing values.

[0089] The present invention provides a data flow method. When a preset scheduling time is reached, a data flow task corresponding to the preset scheduling time is obtained, wherein the data flow task carries a data flow task number; it is determined whether the data flow direction of the data flow task is data extraction or data push; if the data flow direction of the data flow task is data extraction, data extraction configuration information matching the data flow task number is obtained; from a source database indicated by the data extraction configuration information, data in a corresponding source data table is extracted to a corresponding first target data table in a Hadoop database indicated by the data extraction configuration information; a data extraction conversion configuration rule matching the data extraction configuration information is obtained, and a data extraction conversion configuration rule is converted according to the data extraction configuration information. According to the extraction conversion configuration rules and the data extraction configuration information, the first target data table is reorganized; if the data flow direction of the data flow task is data push, the data push configuration information matching the data flow task number and the data push conversion configuration rules matching the data push configuration information are obtained, and according to the data push conversion configuration rules and the data push configuration information, the corresponding second target data table in the Hadoop database indicated by the data push configuration information is reorganized, and the corresponding data is pushed to the corresponding source data table in the source database indicated by the data push information, thereby realizing fast and efficient flow of data between the Hadoop database and the relational database.

[0090] Based on the big data transfer method disclosed in the embodiment of the present invention, the embodiment of the present invention also discloses a data transfer device, such as Figure 2 As shown, the big data circulation device includes:

[0091] The first acquisition unit 21 is used to acquire a data flow task corresponding to the preset scheduling time when the preset scheduling time is reached, wherein the data flow task carries a data flow task number;

[0092] A determination unit 22, used to determine whether the data flow direction of the data flow task is data extraction or data push;

[0093] The second acquisition unit 23 is used to acquire data extraction configuration information matching the data flow task number if the data flow direction of the data flow task is data extraction;

[0094] A data extraction unit 24 is used to extract corresponding data from the source database indicated by the data extraction configuration information to a corresponding first target data table in the Hadoop database indicated by the data extraction configuration information;

[0095] The first data reorganization unit 25 is used to obtain a data extraction conversion configuration rule that matches the data extraction configuration information; and to reorganize the data of the first target data table according to the data extraction conversion configuration rule and the data extraction configuration information;

[0096] The third acquisition unit 26 is used to acquire data push configuration information matching the data flow task number and acquire data push conversion configuration rules matching the data push configuration information if the data flow direction of the data flow task is data push;

[0097] The second data reorganization unit 27 is used to reorganize the data of the corresponding second target data table in the Hadoop database indicated by the data push configuration information according to the data push conversion configuration rule and the data push configuration information, and push the corresponding data to the corresponding source data table in the source database indicated by the data push information.

[0098] The specific principles and execution processes of each unit in the data flow device disclosed in the above embodiment of the present invention are the same as the data flow method disclosed in the above embodiment of the present invention. Please refer to the corresponding parts of the data flow method disclosed in the above embodiment of the present invention, and will not be repeated here.

[0099] The present invention provides a data flow device, which, when a preset scheduling time is reached, obtains a data flow task corresponding to the preset scheduling time, wherein the data flow task carries a data flow task number; determines whether the data flow direction of the data flow task is data extraction or data push; if the data flow direction of the data flow task is data extraction, obtains data extraction configuration information matching the data flow task number; extracts data in a corresponding source data table from a source database indicated by the data extraction configuration information to a corresponding first target data table in a Hadoop database indicated by the data extraction configuration information; obtains a data extraction conversion configuration rule matching the data extraction configuration information, and extracts data according to the data extraction configuration information; ... According to the extraction conversion configuration rules and the data extraction configuration information, the first target data table is reorganized; if the data flow direction of the data flow task is data push, the data push configuration information matching the data flow task number and the data push conversion configuration rules matching the data push configuration information are obtained, and according to the data push conversion configuration rules and the data push configuration information, the corresponding second target data table in the Hadoop database indicated by the data push configuration information is reorganized, and the corresponding data is pushed to the corresponding source data table in the source database indicated by the data push information, thereby realizing fast and efficient flow of data between the Hadoop database and the relational database.

[0100] Optionally, the data extraction configuration information includes database extraction configuration information and data table extraction configuration information, and the data extraction unit includes:

[0101] The data extraction subunit is used to extract data in the source data table indicated by the data table extraction configuration information from the source database indicated by the database extraction configuration information to the first target data table indicated by the data table extraction configuration information in the Hadoop database indicated by the database extraction configuration information.

[0102] Optionally, the first data reassembly unit includes:

[0103] A fourth acquisition unit, used to acquire a data extraction conversion configuration rule matching the data extraction configuration information, wherein the data extraction conversion configuration rule includes a plurality of data processing modes;

[0104] The third data reorganization unit is used to reorganize the data of the first target data table according to various data processing modes and data table extraction configuration information in the data extraction conversion configuration rule.

[0105] Optionally, the data push configuration information includes database push configuration information and data table push configuration information, and the second data reorganization unit includes:

[0106] A fourth data reorganization unit is used to reorganize data on a second target data table indicated by the data table push configuration information in the Hadoop database indicated by the database push configuration information according to various data processing modes indicated by the data push conversion configuration rule and the data table push configuration information;

[0107] The data pushing unit is used to push the data in the second target data table after the data reorganization to the source data table indicated by the data table pushing configuration information in the source database indicated by the database pushing configuration information.

[0108] An embodiment of the present application also provides an electronic device, which includes: a processor and a memory, wherein the processor and the memory are connected via a communication bus; wherein the processor is used to call and execute a program stored in the memory; and the memory is used to store a program, which is used to implement a big data circulation method.

[0109] Reference below Figure 3 , which shows a schematic diagram of the structure of an electronic device suitable for implementing the disclosed embodiment of the present invention. The electronic device in the disclosed embodiment of the present invention may include, but is not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 3 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments disclosed in the present invention.

[0110] like Figure 3 As shown, the electronic device may include a processing device (e.g., a central processing unit, a graphics processor, etc.) 301, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 302 or a program loaded from a storage device 306 into a random access memory (RAM) 303. In RAM 303, various programs and data required for the operation of the electronic device are also stored. The processing device 301, ROM 302, and RAM 303 are connected to each other via a bus 304. An input / output (I / O) interface 305 is also connected to the bus 304.

[0111] Typically, the following devices may be connected to the I / O interface 305: input devices 306 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; output devices 307 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 308 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 309. The communication devices 309 may allow the electronic device to communicate with other devices wirelessly or by wire to exchange data. Although Figure 3 An electronic device having various devices is shown, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0112] In particular, according to the embodiments disclosed in the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present invention include a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 309, or installed from the storage device 308, or installed from the ROM 302. When the computer program is executed by the processing device 301, the above-mentioned functions defined in the method of the embodiment disclosed in the present invention are executed.

[0113] Furthermore, an embodiment of the present invention also provides a computer-readable storage medium, in which computer-executable instructions are stored, and the computer-executable instructions are used to execute the big data circulation method.

[0114] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device: when a preset scheduling time is reached, obtains a data flow task corresponding to the preset scheduling time, wherein the data flow task carries a data flow task number; determines whether the data flow direction of the data flow task is data extraction or data push; if the data flow direction of the data flow task is data extraction, obtains data extraction configuration information matching the data flow task number; extracts corresponding data from a source database indicated by the data extraction configuration information to a corresponding first target data table in a Hadoop database indicated by the data extraction configuration information; obtains Obtain data extraction conversion configuration rules that match the data extraction configuration information; and reorganize the data of the first target data table according to the data extraction conversion configuration rules and the data extraction configuration information; if the data flow direction of the data flow task is data push, obtain the data push configuration information that matches the data flow task number, and obtain the data push conversion configuration rules that match the data push configuration information; according to the data push conversion configuration rules and the data push configuration information, reorganize the data of the corresponding second target data table in the Hadoop database indicated by the data push configuration information, and push its corresponding data to the corresponding source data table in the source database indicated by the data push information.

[0115] In the context disclosed by the present invention, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0116] It should be noted that the computer-readable medium disclosed in the present invention may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present invention, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0117] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

Claims

1. A big data circulation method, characterized in that: The method comprises: When the preset scheduling time is reached, a data flow task corresponding to the preset scheduling time is obtained, wherein the data flow task carries a data flow task number; the data flow task is used to realize data flow between the relational database and the Hadoop database, and the data flow includes data extraction and data push; Determine whether the data flow direction of the data flow task is data extraction or data push; the data extraction is to extract data from the relational database and store it in the Hadoop database, and the data push is to push the data in the Hadoop database to the relational database; If the data flow direction of the data flow task is data extraction, obtaining data extraction configuration information matching the data flow task number; Extracting corresponding data from the source database indicated by the data extraction configuration information to a corresponding first target data table in the Hadoop database indicated by the data extraction configuration information; the data extraction configuration information includes a source database name, a source data table name, a Hadoop database schema, and a target data table name; Acquire a data extraction conversion configuration rule that matches the data extraction configuration information, and reorganize data in the first target data table in the Hadoop database according to the data extraction conversion configuration rule and the data extraction configuration information; wherein, a corresponding data extraction conversion configuration rule is pre-set for each target data table in the Hadoop database; If the data flow direction of the data flow task is data push, obtain data push configuration information matching the data flow task number, and obtain data push conversion configuration rules matching the data push configuration information; wherein corresponding data push conversion configuration rules are pre-set for each data table in the Hadoop database; According to the data push conversion configuration rule and the data push configuration information, data is reorganized for the corresponding second target data table in the Hadoop database indicated by the data push configuration information, and its corresponding data is pushed to the corresponding source data table in the source database indicated by the data push configuration information.

2. The method according to claim 1, characterized in that The data extraction configuration information includes database extraction configuration information and data table extraction configuration information. The step of extracting corresponding data from a source database indicated by the data extraction configuration information to a corresponding first target data table in a Hadoop database indicated by the data extraction configuration information includes: Extract the data in the source data table indicated by the data table extraction configuration information from the source database indicated by the database extraction configuration information to the first target data table indicated by the data table extraction configuration information in the Hadoop database indicated by the database extraction configuration information.

3. The method according to claim 2, characterized in that The step of obtaining a data extraction conversion configuration rule that matches the data extraction configuration information; and reorganizing data in the first target data table according to the data extraction conversion configuration rule and the data extraction configuration information includes: Acquire a data extraction conversion configuration rule that matches the data extraction configuration information, wherein the data extraction conversion configuration rule includes multiple data processing methods; The first target data table is reorganized according to the various data processing modes in the data extraction conversion configuration rules and the data table extraction configuration information.

4. The method according to claim 1, characterized in that: The data push configuration information includes database push configuration information and data table push configuration information, and the step of reorganizing data on a corresponding second target data table in the Hadoop database indicated by the data push configuration information according to the data push conversion configuration rule and the data push configuration information, and pushing the corresponding data to a corresponding source data table in a source database indicated by the data push configuration information, includes: According to various data processing modes indicated by the data push conversion configuration rule and the data table push configuration information, data reorganization is performed on the second target data table indicated by the data table push configuration information in the Hadoop database indicated by the database push configuration information; The data in the second target data table after data reorganization is pushed to the source data table indicated by the data table push configuration information in the source database indicated by the database push configuration information.

5. A big data circulation device, characterized in that: The device comprises: A first acquisition unit is used to acquire a data flow task corresponding to the preset scheduling time when the preset scheduling time is reached, wherein the data flow task carries a data flow task number; the data flow task is used to realize data flow between the relational database and the Hadoop database, and the data flow includes data extraction and data push; A judging unit, used to judge whether the data flow direction of the data flow task is data extraction or data push; the data extraction is to extract data from the relational database and store it in the Hadoop database, and the data push is to push the data in the Hadoop database to the relational database; A second acquisition unit is used to acquire data extraction configuration information matching the data flow task number if the data flow direction of the data flow task is data extraction; the data extraction configuration information includes a source database name, a source data table name, a Hadoop database schema, and a target data table name; A data extraction unit, configured to extract corresponding data from a source database indicated by the data extraction configuration information to a corresponding first target data table in a Hadoop database indicated by the data extraction configuration information; a first data reorganization unit, configured to obtain a data extraction conversion configuration rule matching the data extraction configuration information; and to reorganize data of the first target data table in the Hadoop database according to the data extraction conversion configuration rule and the data extraction configuration information; wherein a corresponding data extraction conversion configuration rule is pre-set for each target data table in the Hadoop database; A third acquisition unit is used to acquire data push configuration information matching the data flow task number if the data flow direction of the data flow task is data push, and acquire data push conversion configuration rules matching the data push configuration information; wherein corresponding data push conversion configuration rules are pre-set for each data table in the Hadoop database; The second data reorganization unit is used to reorganize the data of the corresponding second target data table in the Hadoop database indicated by the data push configuration information according to the data push conversion configuration rule and the data push configuration information, and push the corresponding data to the corresponding source data table in the source database indicated by the data push configuration information.

6. The device according to claim 5, characterized in that The data extraction configuration information includes database extraction configuration information and data table extraction configuration information, and the data extraction unit includes: The data extraction subunit is used to extract data in the source data table indicated by the data table extraction configuration information from the source database indicated by the database extraction configuration information to the first target data table indicated by the data table extraction configuration information in the Hadoop database indicated by the database extraction configuration information.

7. The device according to claim 6, characterized in that The first data reorganization unit includes: A fourth acquisition unit, configured to acquire a data extraction conversion configuration rule matching the data extraction configuration information, wherein the data extraction conversion configuration rule includes a plurality of data processing modes; The third data reorganization unit is used to reorganize the data of the first target data table according to the various data processing modes in the data extraction and conversion configuration rules and the data table extraction configuration information.

8. The device according to claim 5, characterized in that The data push configuration information includes database push configuration information and data table push configuration information, and the second data reorganization unit includes: a fourth data reorganization unit, configured to reorganize data on a second target data table indicated by the data table push configuration information in the Hadoop database indicated by the database push configuration information according to various data processing modes indicated by the data push conversion configuration rule and the data table push configuration information; The data pushing unit is used to push the data in the second target data table after data reorganization to the source data table indicated by the data table pushing configuration information in the source database indicated by the database pushing configuration information.

9. An electronic device, characterized in that: The electronic device includes a processor and a memory, the memory is used to store program codes and data for big data circulation, and the processor is used to call program instructions in the memory to execute the big data circulation method as described in any one of claims 1-4.

10. A storage medium, characterized in that: The storage medium includes a storage program, wherein when the program is running, the device where the storage medium is located is controlled to execute the big data circulation method as described in any one of claims 1-4.

Images