Cross-network service system data ferrying method and system based on ETL, medium and electronic equipment

Abstract

The invention provides an ETL-based cross-network service system data ferrying method and system, a medium and electronic equipment, and the ferrying method comprises the steps: obtaining to-be-ferried data from a network A through employing an ETL technology at regular time, forming a data packet, and storing the data packet in a first path; using the security isolation gatekeeper, the data packet is transmitted to a second path of the network B. And in the network B, acquiring and analyzing the ferry data packet by using an ETL technology, writing data into a receiver system database, generating a feedback data packet according to an execution result (success or failure), and storing the feedback data packet in a third path. The feedback data packet is then transmitted back to a fourth path of the network A, and the ferry result is updated. And if successful, marking as ferry, and if failed, resetting as to-be-ferried and repeating the process. And if the process still fails after being repeated for multiple times, the ferry is marked as'ferry failure ', and ferry is not carried out any more. According to the method provided by the invention, the cross-network service system data can be quickly and accurately ferried, and the problem that the existing cross-network data ferrying efficiency is low is solved.

Description

Technical Field

[0001] The present application relates to the field of computer systems, and more specifically, to an ETL-based cross-network business system data ferrying method, system, medium and electronic device. Background Art

[0002] At present, the IT systems of Chinese enterprises are becoming more and more complete, but many enterprises' IT systems are only built in their own enterprise LANs, and are not connected to the Internet or other enterprise IT systems, resulting in information islands and poor information communication. With the development of information technology and the need for enterprise management reform, more and more enterprises expect to conduct efficient cross-network and cross-enterprise collaboration, which requires a large amount of enterprise intranet data to be transferred to the Internet or other networks, and also requires a large amount of data to be obtained from the Internet or other networks. How to achieve safe, efficient and accurate transfer of data across networks and business systems has always been a problem that enterprises need to solve.

[0003] At present, many enterprises use manual data ferrying, which has high labor costs, poor data timeliness, and is prone to errors, and cannot meet enterprise requirements. After the widespread use of security isolation gateways, many enterprises use customized development services to achieve data ferrying, but the compatibility is poor and the response efficiency is low. Every time the demand changes, development adjustments are required, and it is difficult to monitor the data transmission results, which can easily cause data loss. Therefore, it is urgent to study a cross-network business system data ferrying method and system to quickly and accurately ferry multi-network business system data and promote efficient collaboration between internal and external networks and across enterprises. Summary of the invention

[0004] The purpose of this application is to provide an ETL-based cross-network business system data ferrying method, system, medium and electronic device, which can quickly and accurately ferry cross-network business system data, and solve the problems of low efficiency and easy data loss in existing cross-network data ferrying.

[0005] In a first aspect, a method for transferring data across network business systems based on ETL is provided, including:

[0006] S1, regularly use ETL to continuously obtain the data to be ferried from the data provider system in network A, and store the data to be ferried to form a ferry data packet, which is stored in the first path;

[0007] S2, transmitting the ferry data packet in the first path of network A to the second path of network B through the security isolation network gate and storing it;

[0008] S3, regularly scan the second path in network B, use ETL to obtain the ferry data packet in the second path and parse and read the data, write the read data into the database of the data receiving system of network B, store the execution result data into a feedback data packet, and store it in the third path; the execution result includes failure and success;

[0009] S4, transmitting the feedback data packet in the third path of network B to the fourth path of network A through the security isolation network gate and storing it;

[0010] S5, regularly scan the fourth path of network A, use ETL to obtain the feedback data packet in the fourth path and parse and read the data, write the read data into the database of the data provider system of network A, and update the data ferry result; the data ferry result includes success and failure;

[0011] S6. Analyze the ferrying result in network A and update the data ferrying status; read the data ferrying result, if the ferrying result is successful, set the corresponding data as the ferried data; if the ferrying result is failed, set the corresponding data as the data to be ferried, and repeat steps S1-S6; if the ferrying result is still failed after repeating the predetermined number of times, update the corresponding data status to "ferrying failed" and no longer ferry.

[0012] In one feasible solution, step S1 includes:

[0013] S11, performing system configuration, including configuring the data provider system database connection parameters, the calling frequency of data extraction and output tasks, the ferry data reading statement, the ferry data output statement, and the ferry data packet output path; the ferry data packet output path is the first path in network A;

[0014] S12, start the scheduled task of extracting and outputting data on network A;

[0015] S13. Connect to the database of the data provider system according to the system configuration of S11;

[0016] S14. According to the system configuration of S11, the data to be ferried in the data provider system in network A is read through ETL;

[0017] S15, according to the system configuration of S11, the data read in step S14 is stored in an intermediate data file to form a ferry data packet;

[0018] S16. According to the system configuration of S11, the ferry data packet formed in step S15 is placed in a first path that can be read by the security isolation network gate.

[0019] In an implementable solution, in step S14, when reading the data to be ferried in the data provider system in network A through ETL, an incremental data reading method is adopted.

[0020] In an implementable solution, the step S3 includes:

[0021] S31, perform system configuration, including configuring the connection parameters of the data receiving system database, the calling frequency of the ferry data packet reading and writing tasks, the ferry data packet receiving path, the ferry data packet reading statement, the ferry data packet writing statement, the feedback data packet output path, and the feedback data packet output statement; the feedback data packet output path is the third path in the network B;

[0022] S32, start the scheduled task of ferrying data packets reading and writing in network B;

[0023] S33, according to the system configuration of S31, connecting to the system database of the data receiver;

[0024] S34, according to the system configuration of S31, scan the second path in network B to obtain the ferry data packet;

[0025] S35, according to the system configuration of S31, read the ferry data packet data through ETL;

[0026] S36, according to the system configuration of S31, the read ferry data packet data is stored in the data receiving party system database through ETL;

[0027] S37, according to the system configuration of S31, write the execution result of each data in the ferry data packet into the feedback data packet through ETL;

[0028] S38. According to the system configuration of S31, the feedback data packet formed in step S37 is placed in a third path that can be read by the security isolation network gate.

[0029] In an implementable solution, step S5 includes:

[0030] S51, performing system configuration, including the connection parameters of the business system database of the data provider, the calling frequency of the feedback data packet reading and writing tasks, the feedback data packet reading path, the feedback data packet reading statement, and the feedback data writing statement; the feedback data packet reading path is the fourth path in the network A;

[0031] S52, starting a scheduled task for reading and writing feedback data packets in network A;

[0032] S53, connecting to the system database of the data provider according to the system configuration of S51;

[0033] S54, scanning the fourth path in network A according to the system configuration of S51 to obtain a feedback data packet;

[0034] S55, reading data in the feedback data packet through ETL according to the system configuration of S51;

[0035] S56, according to the system configuration of S51, the read feedback data packet data is stored in the database of the data provider system through ETL, and the data ferry result is updated.

[0036] In an implementable solution, in step S6, if the ferrying result is a failure, the predetermined number of times steps S1-S6 are repeated is set in a range of 3 to 7 times.

[0037] According to the second aspect of the present application, there is also provided an ETL-based cross-network business system data ferry system, including a data provider network, a data receiver network and a security isolation network gate, wherein the data provider network is deployed with a data provider's basic setting module, a scheduled task scheduling module, a data provider's ferry data extraction and output module, and a data provider's feedback data packet reading and writing module;

[0038] The data receiver network is deployed with a data receiver basic setting module, a timed task scheduling module, and a data receiver ferry data packet reading and writing module;

[0039] The basic setting module of the data provider is used to define database connections, modify existing database connections, and delete existing database connections; to define ETL tasks and set call frequencies for them, modify existing ETL tasks, and delete existing ETL tasks; to create, modify, or delete data read statements, data output statements, ferry data packet output paths, feedback data packet read paths, feedback data packet read statements, and feedback data write statements for ETL tasks;

[0040] The basic setting module of the data receiver is used to define database connections, modify existing database connections, and delete existing database connections; to define ETL tasks and set call frequencies for them, modify existing ETL tasks, and delete existing ETL tasks; to create, modify, or delete the ferry data packet receiving path, ferry data packet reading statement, ferry data packet writing statement, feedback data packet output path, and feedback data packet output statement of ETL tasks;

[0041] The scheduled task scheduling module is deployed in both the data provider network and the data receiver network, and is used to automatically and regularly execute data extraction and output tasks, ferry data packet reading and writing tasks, and feedback data packet reading and writing tasks according to the task calling frequency of the basic setting module of the data provider and the task calling frequency set by the basic setting module of the data receiver;

[0042] The ferry data extraction and output module of the data provider is used to establish a database connection according to the database connection information configured by the basic setting module of the data provider, and obtain the data to be ferryed in the data provider's network according to the data reading statement configured by the basic setting module of the data provider, and create a ferry data packet in the specified path according to the ferry data output statement and ferry data packet output path configured by the basic setting module of the data provider, and write the read data into it;

[0043] The security isolation network gate is used to ferry data packets from the data provider network to the data receiver network at a set frequency, and is also used to ferry feedback data packets from the data receiver network to the data provider network;

[0044] The ferry data packet reading and writing module of the data receiver is used to establish a database connection according to the database connection information configured by the basic setting module of the data receiver, and then obtain the ferry data according to the ferry data packet receiving path and the ferry data packet reading statement configured by the basic setting module of the data receiver, and then write the ferry data into the data receiver system database according to the ferry data packet writing statement configured by the basic setting module of the data receiver, and then create a feedback data packet in the specified path according to the feedback data packet output path and the feedback data packet output statement configured by the basic setting module of the data receiver, and write the read data;

[0045] The feedback data packet reading and writing module of the data provider is used to establish a database connection according to the database connection information configured by the basic setting module of the data provider, and then obtain feedback data according to the feedback data packet receiving path and feedback data packet reading statement configured by the basic setting module of the data provider, and then write the feedback data into the data provider database according to the feedback data writing statement configured by the basic setting module of the data provider.

[0046] According to the third aspect of the present application, there is also provided an ETL-based cross-network business system data ferrying system, comprising network A, network B and a security isolation network gate, wherein network A can be used as both a data provider network and a data receiver network; network B can be used as both a data provider network and a data receiver network;

[0047] Network A is deployed with a basic setting module of the data provider, a scheduled task scheduling module, a ferry data extraction and output module of the data provider, a feedback data packet reading and writing module of the data provider, a basic setting module of the data receiver, and a ferry data packet reading and writing module of the data receiver;

[0048] Network B is deployed with a basic setting module of the data provider, a scheduled task scheduling module, a ferry data extraction and output module of the data provider, a feedback data packet reading and writing module of the data provider, a basic setting module of the data receiver, and a ferry data packet reading and writing module of the data receiver;

[0049] The basic setting module of the data provider is used to define database connections, modify existing database connections, and delete existing database connections; to define ETL tasks and set call frequencies for them, modify existing ETL tasks, and delete existing ETL tasks; to create, modify, or delete data read statements, data output statements, ferry data packet output paths, feedback data packet read paths, feedback data packet read statements, and feedback data write statements for ETL tasks;

[0050] The basic setting module of the data receiver is used to define database connections, modify existing database connections, and delete existing database connections; to define ETL tasks and set call frequencies for them, modify existing ETL tasks, and delete existing ETL tasks; to create, modify, or delete the ferry data packet receiving path, ferry data packet reading statement, ferry data packet writing statement, feedback data packet output path, and feedback data packet output statement of ETL tasks;

[0051] A scheduled task scheduling module is used to automatically and regularly execute data extraction and output tasks, ferry data packet reading and writing tasks, and feedback data packet reading and writing tasks according to the task calling frequency of the basic setting module of the data provider and the task calling frequency set by the basic setting module of the data receiver;

[0052] The ferry data extraction and output module of the data provider is used to establish a database connection according to the database connection information configured by the basic setting module of the data provider, and obtain the data to be ferryed in the data provider's network according to the data reading statement configured by the basic setting module of the data provider, and create a ferry data packet in the specified path according to the ferry data output statement and ferry data packet output path configured by the basic setting module of the data provider, and write the read data into it;

[0053] The security isolation network gate is used to ferry data packets from the data provider network to the data receiver network at a set frequency, and is also used to ferry feedback data packets from the data receiver network to the data provider network;

[0054] The ferry data packet reading and writing module of the data receiver is used to establish a database connection according to the database connection information configured by the basic setting module of the data receiver, and then obtain the ferry data according to the ferry data packet receiving path and the ferry data packet reading statement configured by the basic setting module of the data receiver, and then write the ferry data into the data receiver system database according to the ferry data packet writing statement configured by the basic setting module of the data receiver, and then create a feedback data packet in the specified path according to the feedback data packet output path and the feedback data packet output statement configured by the basic setting module of the data receiver, and write the read data.

[0055] The feedback data packet reading and writing module of the data provider is used to establish a database connection according to the database connection information configured by the basic setting module of the data provider, and then obtain feedback data according to the feedback data packet receiving path and feedback data packet reading statement configured by the basic setting module of the data provider, and then write the feedback data into the data provider database according to the feedback data writing statement configured by the basic setting module of the data provider.

[0056] According to a fourth aspect of the present application, a computer-readable storage medium is also provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the aforementioned ETL-based cross-network business system data ferrying method are implemented.

[0057] According to the fifth aspect of the present application, there is also provided an electronic device, including a memory and a processor, wherein the memory stores a computer program, the processor is communicatively connected to the memory, and when the computer program is called, the steps of the aforementioned ETL-based cross-network business system data ferrying method are executed.

[0058] Compared with the prior art, the beneficial effects of this application include at least:

[0059] The ETL-based cross-network business system data ferrying method of the present application is based on ETL (Extract, Transform, Load) technology. This method realizes safe, efficient and accurate data ferrying through automated data extraction, transformation and loading processes, and effectively promotes efficient collaboration between internal and external networks and across enterprises.

[0060] Through the ETL-based cross-network business system data ferrying method and system of this application, the goal of fast and accurate ferrying of cross-network business system data is achieved, solving the problems of low efficiency, easy data loss, and rapid system configuration of existing cross-network data ferrying. The automated ETL process reduces manual intervention and improves the efficiency and accuracy of data ferrying. Data is transmitted through a secure isolation gateway to ensure the security of data during cross-network ferrying. In addition, the method also supports data ferrying between different networks and business systems, has good compatibility and scalability, and can adapt to the ever-changing needs of enterprises. Through the feedback mechanism and result analysis, enterprises can monitor the status of data ferrying in real time, discover and solve problems in a timely manner, thereby providing strong technical support for efficient collaboration between enterprises. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0062] Figure 1 It is a flowchart of a method for transferring data across network business systems based on ETL according to an embodiment of the present application;

[0063] Figure 2 It is a structural diagram of a one-way data ferrying system for cross-network business systems based on ETL according to an embodiment of the present application;

[0064] Figure 3 It is a structural diagram of a cross-network business system data bidirectional ferrying system based on ETL according to an embodiment of the present application;

[0065] Figure 4 It is a schematic diagram of the structure of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0066] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various different configurations.

[0067] Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the present application for which protection is sought, but merely represents selected embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.

[0068] like Figure 1 As shown, the embodiment of the present application first provides a cross-network business system data ferrying method based on ETL, including:

[0069] S1, regularly use ETL to continuously obtain the data to be ferried from the data provider system in network A, and store the data to be ferried to form a ferry data packet, which is stored in the first path;

[0070] S2, transmitting the ferry data packet in the first path of network A to the second path of network B through the security isolation network gate and storing it;

[0071] S3, regularly scan the second path in network B, use ETL to obtain the ferry data packet in the second path and parse and read the data, write the read data into the database of the data receiving system of network B, store the execution result data into a feedback data packet, and store it in the third path; the execution result includes failure and success;

[0072] S4, transmitting the feedback data packet in the third path of network B to the fourth path of network A through the security isolation network gate and storing it;

[0073] S5, regularly scan the fourth path of network A, use ETL to obtain the feedback data packet in the fourth path and parse and read the data, write the read data into the database of the data provider system of network A, and update the data ferry result; the data ferry result includes success and failure;

[0074] S6. Analyze the ferrying result in network A and update the data ferrying status; read the data ferrying result, if the ferrying result is successful, set the corresponding data as the ferried data; if the ferrying result is failed, set the corresponding data as the data to be ferried, and repeat steps S1-S6; if the ferrying result is still failed after repeating the predetermined number of times, update the corresponding data status to "ferrying failed" and no longer ferry.

[0075] It should be noted that ETL technology is the abbreviation of Extract, Transform, and Load. It is an important concept in data warehouse technology. It is used to describe the process of extracting data from the source system (such as relational database, file system, etc.), performing necessary transformation processing, and then loading it into the target system (such as data warehouse or data lake).

[0076] Extraction is the first step in the ETL process. It involves obtaining data from various data sources. These data sources can be structured, such as relational databases, XML files, etc., or unstructured, such as text files, social media data, etc. The extraction process needs to consider the consistency, completeness and accuracy of the data. Transformation is the core step in the ETL process. It involves cleaning, integrating, standardizing and calculating the extracted data. The purpose of transformation is to ensure that the data meets the requirements and business logic of the target system. The transformation process may include data cleaning, data integration, data standardization, and data calculation. Loading is the last step of the ETL process. It involves writing the transformed data to the target system. The loading process needs to consider the storage structure, performance and security of the data. Loading can be batch or real-time, depending on business needs and system design.

[0077] To implement the ETL process, there are a variety of tools and frameworks to choose from, including business intelligence (BI) tools, open source tools, programming languages ​​and libraries, etc. Business intelligence (BI) tools such as Microsoft SQL Server Integration Services (SSIS), Informatica PowerCenter, etc. Open source tools such as Talend, Apache NiFi, Toad, etc. Programming languages ​​and libraries such as Python (using libraries such as Pandas, NumPy, etc.), Java (using JDBC, Hibernate, etc.).

[0078] It should also be noted that a security isolation network gate (also called a network gate or a physical isolation network gate) is a special network security module that is used to achieve secure data exchange between two networks with mutually isolated services. The main feature of this module is that it cuts off the direct connection between the networks at the physical level, thereby ensuring the isolation between the two networks and preventing potential security threats such as hacker attacks, virus propagation, etc. The data ferrying process of the security isolation network gate usually involves data extraction, security checks, data isolation, data transmission, data output, etc. Throughout the process, the security isolation network gate ensures that the data will not be accessed or tampered with by unauthorized third parties during transmission. The design of the security isolation network gate usually includes an intranet processing unit*, an extranet processing unit, an isolation and exchange control unit, etc.

[0079] The ETL-based cross-network business system data ferrying method of this embodiment is based on ETL (Extract, Transform, Load) technology. This method realizes safe, efficient and accurate data ferrying through automated data extraction, transformation and loading processes, and effectively promotes efficient collaboration between internal and external networks and across enterprises.

[0080] Specifically, the method first obtains the data to be ferryed from the data provider system in network A through the ETL technology at regular intervals, and packages and stores the data in the first path. Subsequently, the ferry data packet in the first path is safely transmitted to the second path of network B by using the security isolation network gate, and stored. In network B, the ferry data packet is obtained by using the ETL technology again through the regular scanning of the second path, and the data is parsed and read, and the data is written into the database of the data receiving system. At the same time, the execution result (success or failure) is stored to form a feedback data packet and stored in the third path. Then, the feedback data packet in the third path is transmitted to the fourth path of network A through the security isolation network gate, and the feedback of the ferrying result is provided to the data provider system. In network A, the feedback data packet is obtained and the data is parsed and read by regular scanning of the fourth path, and the data ferrying result is updated. According to the ferrying result, the data status is updated. If the ferrying is successful, the data is marked as ferried; if the ferrying fails, it is reset to the data to be ferryed, and the above steps are repeated. If the ferrying fails more than a predetermined number of times, the data status is updated to "ferrying failure", and no ferrying is performed.

[0081] By implementing this technical solution, this patent achieves the goal of quickly and accurately ferrying data across network business systems, solving the existing problems of low efficiency, easy data loss, and rapid system configuration in cross-network data ferrying. The automated ETL process reduces manual intervention and improves the efficiency and accuracy of data ferrying. Data is transmitted through a secure isolation gateway to ensure the security of data during cross-network ferrying. In addition, this method also supports data ferrying between different networks and business systems, has good compatibility and scalability, and can adapt to the ever-changing needs of enterprises. Through feedback mechanisms and result analysis, enterprises can monitor the status of data ferrying in real time, discover and solve problems in a timely manner, and thus provide strong technical support for efficient collaboration between enterprises.

[0082] In one feasible solution, step S1 may include:

[0083] S11, performing system configuration, including configuring the data provider system database connection parameters, the calling frequency of data extraction and output tasks, the ferry data reading statement, the ferry data output statement, and the ferry data packet output path; the ferry data packet output path is the first path in network A;

[0084] S12, start the scheduled task of extracting and outputting data on network A;

[0085] S13. Connect to the database of the data provider system according to the system configuration of S11;

[0086] S14. According to the system configuration of S11, the data to be ferried in the data provider system in network A is read through ETL;

[0087] S15. According to the system configuration of S11, the data read in step S14 is stored in an intermediate data file. The intermediate data file package format can be XML, excel, txt, SQLite, etc., to form a ferry data package;

[0088] S16. According to the system configuration of S11, the ferry data packet formed in step S15 is placed in a first path that can be read by the security isolation network gate.

[0089] In one feasible solution, in step S14, when reading the data to be ferried in the data provider system in network A through ETL, an incremental data reading method can be adopted, usually adding a timestamp or transfer status to the data table to be ferried in the business system database, thereby facilitating incremental data reading.

[0090] In an implementable solution, the step S3 may include:

[0091] S31, perform system configuration, including configuring the connection parameters of the data receiving system database, the calling frequency of the ferry data packet reading and writing tasks, the ferry data packet receiving path, the ferry data packet reading statement, the ferry data packet writing statement, the feedback data packet output path, and the feedback data packet output statement; the feedback data packet output path is the third path in the network B;

[0092] S32, start the scheduled task of ferrying data packets reading and writing in network B;

[0093] S33, according to the system configuration of S31, connecting to the system database of the data receiver;

[0094] S34, according to the system configuration of S31, scan the second path in network B to obtain the ferry data packet;

[0095] S35, according to the system configuration of S31, read the ferry data packet data through ETL;

[0096] S36, according to the system configuration of S31, the read ferry data packet data is stored in the data receiving party system database through ETL;

[0097] S37, according to the system configuration of S31, write the execution result of each data in the ferry data packet into the feedback data packet through ETL;

[0098] S38. According to the system configuration of S31, the feedback data packet formed in step S37 is placed in a third path that can be read by the security isolation network gate.

[0099] In an practicable solution, step S5 may include:

[0100] S51, performing system configuration, including the connection parameters of the business system database of the data provider, the calling frequency of the feedback data packet reading and writing tasks, the feedback data packet reading path, the feedback data packet reading statement, and the feedback data writing statement; the feedback data packet reading path is the fourth path in the network A;

[0101] S52, starting a scheduled task for reading and writing feedback data packets in network A;

[0102] S53, connecting to the system database of the data provider according to the system configuration of S51;

[0103] S54, scanning the fourth path in network A according to the system configuration of S51 to obtain a feedback data packet;

[0104] S55, reading data in the feedback data packet through ETL according to the system configuration of S51;

[0105] S56, according to the system configuration of S51, the read feedback data packet data is stored in the database of the data provider system through ETL, and the data ferry result is updated.

[0106] In an practicable solution, in step S6, if the ferrying result is a failure, the predetermined number of times of repeating steps S1-S6 can be set in a range of 3 to 7 times, for example, 3 times, 4 times, 5 times, 6 times, 7 times, etc. The number of repeated ferrying can be reasonably set according to the time consumption of each ferrying and the amount of data.

[0107] like Figure 2 As shown, the embodiment of the present application provides a cross-network business system data ferry system based on ETL, including a data provider network, a data receiver network and a security isolation network gate, wherein the data provider network is deployed with a data provider's basic setting module, a timed task scheduling module, a data provider's ferry data extraction and output module, and a data provider's feedback data packet reading and writing module;

[0108] The data receiver network is deployed with a data receiver basic setting module, a timed task scheduling module, and a data receiver ferry data packet reading and writing module;

[0109] The basic setting module of the data provider is used to define database connections, modify existing database connections, and delete existing database connections; to define ETL tasks and set call frequencies for them, modify existing ETL tasks, and delete existing ETL tasks; to create, modify, or delete data read statements, data output statements, ferry data packet output paths, feedback data packet read paths, feedback data packet read statements, and feedback data write statements for ETL tasks;

[0110] The basic setting module of the data receiver is used to define database connections, modify existing database connections, and delete existing database connections; to define ETL tasks and set call frequencies for them, modify existing ETL tasks, and delete existing ETL tasks; to create, modify, or delete the ferry data packet receiving path, ferry data packet reading statement, ferry data packet writing statement, feedback data packet output path, and feedback data packet output statement of ETL tasks;

[0111] The scheduled task scheduling module is deployed in both the data provider network and the data receiver network, and is used to automatically and regularly execute data extraction and output tasks, ferry data packet reading and writing tasks, and feedback data packet reading and writing tasks according to the task calling frequency of the basic setting module of the data provider and the task calling frequency set by the basic setting module of the data receiver;

[0112] The ferry data extraction and output module of the data provider is used to establish a database connection according to the database connection information configured by the basic setting module of the data provider, and obtain the data to be ferryed in the data provider's network according to the data reading statement configured by the basic setting module of the data provider, and create a ferry data packet in the specified path according to the ferry data output statement and ferry data packet output path configured by the basic setting module of the data provider, and write the read data into it;

[0113] The security isolation network gate is used to ferry data packets from the data provider network to the data receiver network at a set frequency, and is also used to ferry feedback data packets from the data receiver network to the data provider network;

[0114] The ferry data packet reading and writing module of the data receiver is used to establish a database connection according to the database connection information configured by the basic setting module of the data receiver, and then obtain the ferry data according to the ferry data packet receiving path and the ferry data packet reading statement configured by the basic setting module of the data receiver, and then write the ferry data into the data receiver system database according to the ferry data packet writing statement configured by the basic setting module of the data receiver, and then create a feedback data packet in the specified path according to the feedback data packet output path and the feedback data packet output statement configured by the basic setting module of the data receiver, and write the read data;

[0115] The feedback data packet reading and writing module of the data provider is used to establish a database connection according to the database connection information configured by the basic setting module of the data provider, and then obtain feedback data according to the feedback data packet receiving path and feedback data packet reading statement configured by the basic setting module of the data provider, and then write the feedback data into the data provider database according to the feedback data writing statement configured by the basic setting module of the data provider.

[0116] like Figure 3 As shown, the embodiment of the present application also provides a cross-network business system data ferrying system based on ETL, including network A, network B and a security isolation network gate, network A can be used as both a data provider network and a data receiver network; network B can be used as both a data provider network and a data receiver network;

[0117] Network A is deployed with a basic setting module of the data provider, a scheduled task scheduling module, a ferry data extraction and output module of the data provider, a feedback data packet reading and writing module of the data provider, a basic setting module of the data receiver, and a ferry data packet reading and writing module of the data receiver;

[0118] Network B is deployed with a basic setting module of the data provider, a scheduled task scheduling module, a ferry data extraction and output module of the data provider, a feedback data packet reading and writing module of the data provider, a basic setting module of the data receiver, and a ferry data packet reading and writing module of the data receiver;

[0119] The basic setting module of the data provider is used to define database connections, modify existing database connections, and delete existing database connections; to define ETL tasks and set call frequencies for them, modify existing ETL tasks, and delete existing ETL tasks; to create, modify, or delete data read statements, data output statements, ferry data packet output paths, feedback data packet read paths, feedback data packet read statements, and feedback data write statements for ETL tasks;

[0120] The basic setting module of the data receiver is used to define database connections, modify existing database connections, and delete existing database connections; to define ETL tasks and set call frequencies for them, modify existing ETL tasks, and delete existing ETL tasks; to create, modify, or delete the ferry data packet receiving path, ferry data packet reading statement, ferry data packet writing statement, feedback data packet output path, and feedback data packet output statement of ETL tasks;

[0121] A scheduled task scheduling module is used to automatically and regularly execute data extraction and output tasks, ferry data packet reading and writing tasks, and feedback data packet reading and writing tasks according to the task calling frequency of the basic setting module of the data provider and the task calling frequency set by the basic setting module of the data receiver;

[0122] The ferry data extraction and output module of the data provider is used to establish a database connection according to the database connection information configured by the basic setting module of the data provider, and obtain the data to be ferryed in the data provider's network according to the data reading statement configured by the basic setting module of the data provider, and create a ferry data packet in the specified path according to the ferry data output statement and ferry data packet output path configured by the basic setting module of the data provider, and write the read data into it;

[0123] The security isolation network gate is used to ferry data packets from the data provider network to the data receiver network at a set frequency, and is also used to ferry feedback data packets from the data receiver network to the data provider network;

[0124] The ferry data packet reading and writing module of the data receiver is used to establish a database connection according to the database connection information configured by the basic setting module of the data receiver, and then obtain the ferry data according to the ferry data packet receiving path and the ferry data packet reading statement configured by the basic setting module of the data receiver, and then write the ferry data into the data receiver system database according to the ferry data packet writing statement configured by the basic setting module of the data receiver, and then create a feedback data packet in the specified path according to the feedback data packet output path and the feedback data packet output statement configured by the basic setting module of the data receiver, and write the read data.

[0125] The feedback data packet reading and writing module of the data provider is used to establish a database connection according to the database connection information configured by the basic setting module of the data provider, and then obtain feedback data according to the feedback data packet receiving path and feedback data packet reading statement configured by the basic setting module of the data provider, and then write the feedback data into the data provider database according to the feedback data writing statement configured by the basic setting module of the data provider.

[0126] Figure 3 The cross-network business system data ferrying system based on ETL shown in the figure deploys the basic setting module of the data provider, the scheduled task scheduling module, the ferrying data extraction and output module of the data provider, the feedback data packet reading and writing module of the data provider, the basic setting module of the data receiver, and the ferrying data packet reading and writing module of the data receiver on both network A and network B. This deployment method can realize the two-way ferrying of business system data between network A and network B.

[0127] This embodiment also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, all or part of the steps of the ETL-based cross-network business system data ferrying method in the aforementioned solution are implemented.

[0128] It will be understood by a person skilled in the art that all or part of the steps in the method for implementing the above-mentioned embodiment can be completed by a program instruction processor, and the program can be stored in a computer-readable storage medium, and the storage medium can be a non-transitory medium, such as a random access memory, a read-only memory, a flash memory, a hard disk, a solid-state hard disk, a magnetic tape, a floppy disk, an optical disc, and any combination thereof. The above-mentioned storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or a data center that includes one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a digital video disc (DVD)), or a semiconductor medium (e.g., a solid-state disk (SSD)), etc.

[0129] like Figure 4 As shown, the present application also provides an electronic device 100, which includes a memory 101 and a processor 102. The memory 101 stores a computer program. The processor 102 is connected to the memory 101 for communication, and executes all or part of the steps of the cross-network business system data ferrying method based on ETL in the above-mentioned scheme when calling the computer program.

[0130] In some possible implementations, the memory 101 may include various media that can store program codes, such as ROM, RAM, magnetic disk, U disk, memory card or optical disk. Specifically, the memory 101 may include a computer system readable medium in the form of a volatile memory, such as a random access memory (RAM) and / or a cache memory. The electronic device 100 may further include other removable / non-removable, volatile / non-volatile computer system storage media. The memory 101 may include at least one program product, which has a group (e.g., at least one) of program modules, which are configured to perform the functions of various embodiments of the present invention. The processor 102 may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components.

[0131] The above description is only the preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A cross-network business system data ferrying method based on ETL, It is characterized in that include: S1, regularly use ETL to continuously obtain the data to be ferried from the data provider system in network A, and store the data to be ferried to form a ferry data packet, which is stored in the first path; S2, transmitting the ferry data packet in the first path of network A to the second path of network B through the security isolation network gate and storing it; S3, regularly scan the second path in network B, use ETL to obtain the ferry data packet in the second path and parse and read the data, write the read data into the database of the data receiving system of network B, store the execution result data into a feedback data packet, and store it in the third path; the execution result includes failure and success; S4, transmitting the feedback data packet in the third path of network B to the fourth path of network A through the security isolation network gate and storing it; S5, regularly scan the fourth path of network A, use ETL to obtain the feedback data packet in the fourth path and parse and read the data, write the read data into the database of the data provider system of network A, and update the data ferry result; the data ferry result includes success and failure; S6. Analyze the ferrying result in network A and update the data ferrying status; read the data ferrying result, if the ferrying result is successful, set the corresponding data as the ferried data; if the ferrying result is failed, set the corresponding data as the data to be ferryed, and repeat steps S1-S6; if the ferrying result is still failed after repeating the predetermined number of times, update the corresponding data status to "ferrying failed" and no longer ferry.

2. According to claim 1, the cross-network business system data ferrying method based on ETL, It is characterized in that Step S1 includes: S11, performing system configuration, including configuring the data provider system database connection parameters, the calling frequency of data extraction and output tasks, the ferry data reading statement, the ferry data output statement, and the ferry data packet output path; the ferry data packet output path is the first path in network A; S12, start the scheduled task of extracting and outputting data on network A; S13. Connect to the database of the data provider system according to the system configuration of S11; S14. According to the system configuration of S11, the data to be ferried in the data provider system in network A is read through ETL; S15, according to the system configuration of S11, the data read in step S14 is stored in an intermediate data file to form a ferry data packet; S16. According to the system configuration of S11, the ferry data packet formed in step S15 is placed in a first path that can be read by the security isolation network gate.

3. According to claim 2, the cross-network business system data ferrying method based on ETL, It is characterized in that In step S14, when reading the data to be ferried in the data provider system in network A through ETL, an incremental data reading method is adopted.

4. The cross-network business system data ferrying method based on ETL according to claim 1, It is characterized in that The step S3 comprises: S31, perform system configuration, including configuring the connection parameters of the data receiving system database, the calling frequency of the ferry data packet reading and writing tasks, the ferry data packet receiving path, the ferry data packet reading statement, the ferry data packet writing statement, the feedback data packet output path, and the feedback data packet output statement; the feedback data packet output path is the third path in the network B; S32, start the scheduled task of ferrying data packets reading and writing in network B; S33, according to the system configuration of S31, connecting to the system database of the data receiver; S34, according to the system configuration of S31, scan the second path in network B to obtain the ferry data packet; S35, according to the system configuration of S31, read the ferry data packet data through ETL; S36, according to the system configuration of S31, the read ferry data packet data is stored in the data receiving party system database through ETL; S37, according to the system configuration of S31, write the execution result of each data in the ferry data packet into the feedback data packet through ETL; S38. According to the system configuration of S31, the feedback data packet formed in step S37 is placed in a third path that can be read by the security isolation network gate.

5. The cross-network business system data ferrying method based on ETL according to claim 1, It is characterized in that The step S5 comprises: S51, performing system configuration, including the connection parameters of the business system database of the data provider, the calling frequency of the feedback data packet reading and writing tasks, the feedback data packet reading path, the feedback data packet reading statement, and the feedback data writing statement; the feedback data packet reading path is the fourth path in the network A; S52, starting a scheduled task for reading and writing feedback data packets in network A; S53, connecting to the system database of the data provider according to the system configuration of S51; S54, scanning the fourth path in network A according to the system configuration of S51 to obtain a feedback data packet; S55, reading data in the feedback data packet through ETL according to the system configuration of S51; S56, according to the system configuration of S51, the read feedback data packet data is stored in the database of the data provider system through ETL, and the data ferry result is updated.

6. The cross-network business system data ferrying method based on ETL according to claim 1, It is characterized in that In step S6, if the ferrying result is a failure, the predetermined number of times steps S1-S6 are repeated is set in the range of 3 to 7 times.

7. A cross-network business system data ferrying system based on ETL, including a data provider network, a data receiver network and a security isolation network gate. It is characterized in that The data provider network is deployed with the data provider's basic setting module, scheduled task scheduling module, data provider's ferry data extraction and output module, and data provider's feedback data packet reading and writing module; The data receiver network is deployed with a data receiver basic setting module, a timed task scheduling module, and a data receiver ferry data packet reading and writing module; The basic setting module of the data provider is used to define database connections, modify existing database connections, and delete existing database connections; it is used to define ETL tasks and set the calling frequency for them, modify existing ETL tasks, and delete existing ETL tasks; Used to create, modify or delete data read statements, data output statements, ferry data packet output paths, feedback data packet read paths, feedback data packet read statements, and feedback data write statements for ETL tasks; The basic setting module of the data receiver is used to define database connections, modify existing database connections, and delete existing database connections; it is used to define ETL tasks and set call frequencies for them, modify existing ETL tasks, and delete existing ETL tasks; Used to create, modify or delete the ferry data packet receiving path, ferry data packet reading statement, ferry data packet writing statement, feedback data packet output path, feedback data packet output statement of the ETL task; The scheduled task scheduling module is deployed in both the data provider network and the data receiver network, and is used to automatically and regularly execute data extraction and output tasks, ferry data packet reading and writing tasks, and feedback data packet reading and writing tasks according to the task calling frequency of the basic setting module of the data provider and the task calling frequency set by the basic setting module of the data receiver; The ferry data extraction and output module of the data provider is used to establish a database connection according to the database connection information configured by the basic setting module of the data provider, and to obtain the data to be ferryed in the data provider's network according to the data reading statement configured by the basic setting module of the data provider, and to create a ferry data packet in the specified path according to the ferry data output statement and ferry data packet output path configured by the basic setting module of the data provider, and write the read data into the ferry data packet; The security isolation network gate is used to ferry data packets from the data provider network to the data receiver network at a set frequency, and is also used to ferry feedback data packets from the data receiver network to the data provider network; The ferry data packet reading and writing module of the data receiver is used to establish a database connection according to the database connection information configured by the basic setting module of the data receiver, and then obtain the ferry data according to the ferry data packet receiving path and the ferry data packet reading statement configured by the basic setting module of the data receiver, and then write the ferry data into the data receiver system database according to the ferry data packet writing statement configured by the basic setting module of the data receiver, and then create a feedback data packet in the specified path according to the feedback data packet output path and the feedback data packet output statement configured by the basic setting module of the data receiver, and write the read data into it; The feedback data packet reading and writing module of the data provider is used to establish a database connection according to the database connection information configured by the basic setting module of the data provider, then obtain feedback data according to the feedback data packet receiving path and feedback data packet reading statement configured by the basic setting module of the data provider, and then write the feedback data into the data provider database according to the feedback data writing statement configured by the basic setting module of the data provider.

8. A cross-network business system data ferrying system based on ETL, including network A, network B and security isolation network gate, It is characterized in that The network A can be used as both a data provider network and a data receiver network; the network B can be used as both a data provider network and a data receiver network; Network A is deployed with a basic setting module of the data provider, a scheduled task scheduling module, a ferry data extraction and output module of the data provider, a feedback data packet reading and writing module of the data provider, a basic setting module of the data receiver, and a ferry data packet reading and writing module of the data receiver; Network B is deployed with a basic setting module of the data provider, a scheduled task scheduling module, a ferry data extraction and output module of the data provider, a feedback data packet reading and writing module of the data provider, a basic setting module of the data receiver, and a ferry data packet reading and writing module of the data receiver; The basic setting module of the data provider is used to define database connections, modify existing database connections, and delete existing database connections; it is used to define ETL tasks and set the calling frequency for them, modify existing ETL tasks, and delete existing ETL tasks; Used to create, modify or delete data read statements, data output statements, ferry data packet output paths, feedback data packet read paths, feedback data packet read statements, and feedback data write statements for ETL tasks; The basic setting module of the data receiver is used to define database connections, modify existing database connections, and delete existing database connections; it is used to define ETL tasks and set call frequencies for them, modify existing ETL tasks, and delete existing ETL tasks; Used to create, modify or delete the ferry data packet receiving path, ferry data packet reading statement, ferry data packet writing statement, feedback data packet output path, feedback data packet output statement of the ETL task; The scheduled task scheduling module is used to automatically and regularly execute data extraction and output tasks, ferry data packet reading and writing tasks, and feedback data packet reading and writing tasks according to the task calling frequency of the basic setting module of the data provider and the task calling frequency set by the basic setting module of the data receiver; The ferry data extraction and output module of the data provider is used to establish a database connection according to the database connection information configured by the basic setting module of the data provider, and to obtain the data to be ferryed in the data provider's network according to the data reading statement configured by the basic setting module of the data provider, and to create a ferry data packet in the specified path according to the ferry data output statement and ferry data packet output path configured by the basic setting module of the data provider, and write the read data into the ferry data packet; The security isolation network gate is used to ferry data packets from the data provider network to the data receiver network at a set frequency, and is also used to ferry feedback data packets from the data receiver network to the data provider network; The ferry data packet reading and writing module of the data receiver is used to establish a database connection according to the database connection information configured by the basic setting module of the data receiver, and then obtain the ferry data according to the ferry data packet receiving path and the ferry data packet reading statement configured by the basic setting module of the data receiver, and then write the ferry data into the data receiver system database according to the ferry data packet writing statement configured by the basic setting module of the data receiver, and then create a feedback data packet in the specified path according to the feedback data packet output path and the feedback data packet output statement configured by the basic setting module of the data receiver, and write the read data into it; The feedback data packet reading and writing module of the data provider is used to establish a database connection according to the database connection information configured by the basic setting module of the data provider, then obtain feedback data according to the feedback data packet receiving path and feedback data packet reading statement configured by the basic setting module of the data provider, and then write the feedback data into the data provider database according to the feedback data writing statement configured by the basic setting module of the data provider.

9. A computer-readable storage medium having a computer program stored thereon, It is characterized in that When the computer program is executed by a processor, the steps of the cross-network business system data ferrying method based on ETL as described in any one of claims 1 to 6 are implemented.

10. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, and the processor is communicatively connected to the memory. It is characterized in that When the computer program is called, the steps of the cross-network business system data ferrying method based on ETL as described in any one of claims 1 to 6 are executed.

Images