A method, system and storage medium for realizing cross-network data synchronization using a scheduling algorithm based on an isolation device

Abstract

A method, system, and storage medium for achieving cross-network data synchronization using a scheduling algorithm based on an isolation device include: activating an archive log to capture change records of data table objects, parsing the archive log into standard SQL statements; when data in a general structured table is changed, SQL statements generated by parsing the archive log are written into a change record table; for structured tables that record file storage paths, records are recorded in a record table; scheduling service programs A and B are deployed on an extranet server to respectively execute SQL statements for structured data changes and SQL statements for structured data changes in an intranet database; scheduling service program C is deployed on an intranet server to read records in the record table, generate local temporary unstructured files from file streams, and then upload the file data to an intranet file server. The present invention can ensure the consistency of data across network zones and improve the availability and reliability of intranet and extranet business systems within an industry.

Description

Technical Field

[0001] The present invention relates to the field of electronic information technology, and in particular to a method, system and storage medium for realizing cross-network data synchronization using a scheduling algorithm based on an isolation device. Background Art

[0002] The network environment of State Grid Corporation of China is divided into the Internet zone and the management information zone. Usually, some application systems need to be used in the Internet zone and the management information zone at the same time. At the same time, the structured and unstructured data generated in the Internet zone need to be accessed in the intranet application system in near real time, and vice versa.

[0003] The isolation devices used in production environments are typically SQL proxy-type. External networks access the intranet database through the isolation device, and business systems use SG-JDBC via a secure custom protocol to access the isolation device's proxy server. The proxy server accesses data by calling SQL statements. However, when encountering large unstructured files, directly storing stream data using BLOB fields in the intranet management zone database is not only inefficient and occupies a large amount of database storage space, but also has a negative impact on database I / O pressure, memory overhead, and overall SQL performance. Therefore, there is an urgent need to use existing isolation devices to leverage structured data tags to synchronize unstructured data across network zones. Summary of the Invention

[0004] The present invention provides a method, system and storage medium for realizing cross-network data synchronization using a scheduling algorithm based on an isolation device to solve the above-mentioned technical problems. The method uses an isolation device to quickly and accurately realize the synchronization of structured and unstructured data, ensure the consistency of cross-network data, and improve the availability and reliability of intranet and extranet business systems.

[0005] A method for implementing cross-network data synchronization based on an isolation device using a scheduling algorithm includes the following steps:

[0006] Step 1: Configure the external database, enable archive logs to capture changes to table objects, and develop a log parsing service to parse archive logs into standard SQL statements.

[0007] Step 2: When the user generates new structured and unstructured data in the external network operating system, the unstructured data is stored on the file server, and the file storage path is stored in the structured table. When the general structured table data is changed, the archive log is parsed to generate SQL statements and written into the table T_DATA_W (external network structured data recording table). The file storage path is recorded using the table T_FILE_W (external network unstructured path recording table), and the structured and unstructured records are stored and processed separately.

[0008] Step 3: Deploy scheduling service program A on the external network server and read the data recorded in table T_DATA_W (external network structured data recording table). T_DATA_W (external network structured data recording table) records the SQL statements for structured data changes. Leveraging the characteristics of the isolation device, the SQL statements are directly executed in the internal network database through the isolation device proxy to achieve structured data synchronization.

[0009] Step 4: Deploy the scheduling service program B on the external network server, read the file storage path recorded in T_FILE_W (external network unstructured path record table), read the file path to generate a file stream and write it into the SQL statement variable, execute the SQL statement storing the file stream information directly in the internal network database through the change record isolation device agent, insert it into T_FILE_N (intranet unstructured file stream information record table) for storage, and realize cross-region synchronization of file data;

[0010] Step 5: Deploy the scheduling service program C on the intranet server, read the records in T_FILE_N (intranet unstructured file stream information record table), and generate a local temporary unstructured file from the file stream in the scheduling service program. Then upload the file data to the intranet file server, and finally complete the file synchronization;

[0011] Step 6: After the synchronization of structured data and unstructured data is completed, delete the synchronized records in the T_DATA_W, T_FILE_W, and T_FILE_N tables, and delete the temporary unstructured files to release resources.

[0012] Furthermore, the network security isolation device is arranged between the information intranet and the information extranet, and is used to achieve high-intensity logical isolation of the information intranet and extranet systems and devices, and use SQL proxy to achieve data exchange.

[0013] Furthermore, the archive log is an inactive redo log backup of the database, and all redo history records are retained by using the archive log.

[0014] A system for implementing cross-network data synchronization using a scheduling algorithm based on an isolation device includes: a computer-readable storage medium and a processor;

[0015] The computer-readable storage medium is used to store executable instructions;

[0016] The processor is used to read the executable instructions stored in the computer-readable storage medium and execute the method for implementing cross-network data synchronization based on the isolation device using a scheduling algorithm.

[0017] A non-transitory computer-readable storage medium stores a computer program, which, when executed by a processor, implements the method for realizing cross-network data synchronization based on an isolation device using a scheduling algorithm.

[0018] The present invention achieves strong isolation between the internal and external information networks of the State Grid Corporation of China through the deployment and use of isolation devices, effectively resisting the destruction and attack activities of malicious codes, hackers, etc. from the Internet and external networks on the company's information internal network. The use of a scheduling algorithm based on the isolation device to achieve cross-network data synchronization can solve the external-to-internal synchronization and internal-to-external synchronization of a large amount of data, including structured and unstructured data. During the synchronization process, the scheduling algorithm is used to ensure the quasi-real-time performance of data synchronization, meeting the goal of no distinction and no delay in the application data of the internal and external network systems. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 It is a working principle diagram of the present invention;

[0020] Figure 2 It is a data flow diagram of the present invention;

[0021] Figure 3 This is a flowchart of a method for realizing cross-network data synchronization based on an isolation device using a scheduling algorithm according to an embodiment of the present invention. DETAILED DESCRIPTION

[0022] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0023] like Figure 1 and Figure 2 As shown, information intranet and information extranet: the network environment of State Grid Corporation is divided into the Internet area and the management information area. The Internet area is the information extranet, and the management information area is the information intranet; network security isolation device: realizes high-intensity logical isolation of information intranet and intranet systems and equipment, and uses SQL proxy to realize data exchange.

[0024] The present invention implements this solution from three aspects: internal and external network database environment configuration, file data and corresponding structured data reading and writing processing, and scheduling service processing. The specific description is as follows:

[0025] Internal and external database environment configuration: First, configure the external database archive log environment variables to capture changes to structured data. Next, develop a database log parsing service to parse SQL statements in the logs and cache them.

[0026] Read and write file data and corresponding structured data: After the application service obtains structured data, it is stored in the database server. When file data is obtained, it is stored in the file server.

[0027] Scheduling service processing: During one-way synchronization from external to internal, the external network scheduling service reads the cached SQL change record and executes the SQL statement in the intranet database through the SQL proxy function of the isolation device; for unstructured files, the scheduling service reads the file stream and writes it to the BLOB field of the temporary table of the intranet database; the intranet scheduling service reads the file stream information from the BLOB field of the temporary table of the intranet database, and restores it to the intranet file server on the intranet, and records the file-related information to the intranet database, and then provides the intranet application service to read the file to achieve file data synchronization. Conversely, during one-way synchronization from internal to external, the principle is the same.

[0028] The archive log is an inactive redo log backup of the database. By using the archive log, all redo history records can be retained.

[0029] The above are the key technical points in the process of data synchronization from the external network to the internal network. Conversely, the technical points for synchronization from the internal network to the external network are the same.

[0030] Combine Figure 3 As shown, the first aspect of the present invention provides a method for implementing cross-network data synchronization based on an isolation device using a scheduling algorithm, comprising the following steps:

[0031] Step 1: Configure the external database, start the archive log to capture the change records of the data table objects, develop the log parsing service, and parse the archive log into standard SQL statements.

[0032] Step 2: When the user generates new structured and unstructured data in the external network operating system, the unstructured data is stored in Figure 1 On the file server, the file storage path is stored in a structured table. When the data of a general structured table is changed, the archive log is parsed to generate an SQL statement and written into the change record table T_DATA_TCN. For the structured table that records the file storage path, the record table T_FILE_TCN is used to record it, and the structured and unstructured records are stored and processed separately.

[0033] Step 3: Deploy the scheduling service program A on the external network server and read the data recorded in the change record table T_DATA_TCN. This table records the SQL statements for structured data changes. Utilizing the characteristics of the isolation device itself, the SQL statements are executed directly in the database of the intranet through the isolation device agent to achieve structured data synchronization.

[0034] Step 4: Deploy the scheduling service program B on the external network server, read the file storage path recorded in the change record table T_FILE_TCN, read the file path to generate a file stream and write it into the SQL statement variable, and again use the characteristics of the isolation device itself to execute the SQL statement that stores the file stream information directly in the intranet database through the isolation device agent, and insert it into the intranet database table T_FILE_DATA for storage, which is a key step in realizing cross-region synchronization of file data.

[0035] Step 5: Deploy the scheduling service program C on the intranet server, read the records in the record table T_FILE_DATA of the file stream data written from the external network, and generate a local temporary unstructured file from the file stream in the scheduling service program, then upload the file data to the intranet file server, and finally complete the file synchronization.

[0036] Step 6: After the synchronization of structured data and unstructured data is completed, delete the records in the synchronized structured data tables T_DATA_TCN, T_FILE_TCN, and T_FILE_DATA, and delete the temporary unstructured files to release the resources.

[0037] The present invention realizes instant cross-region file synchronization through an isolation device, solves the difficulty of displaying files across regions, and avoids various disadvantages brought about by using a database to store binary streams of files, such as wasting database storage resources, greatly consuming system IO resources, unstable cross-region reading streams, and performance issues. By utilizing the characteristics of the database and the isolation device, a high-speed data synchronization service is developed to realize rapid synchronization of structured and unstructured data, and a scheduling algorithm is used to improve data synchronization efficiency.

[0038] Another aspect of the present invention provides a system for implementing cross-network data synchronization using a scheduling algorithm based on an isolation device, comprising: a computer-readable storage medium and a processor;

[0039] The computer-readable storage medium is used to store executable instructions;

[0040] The processor is used to read the executable instructions stored in the computer-readable storage medium and execute the method for implementing cross-network data synchronization based on the isolation device using a scheduling algorithm as described in the first aspect.

[0041] On the other hand, the present invention provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the method for realizing cross-network data synchronization based on the isolation device using a scheduling algorithm as described in the first aspect is implemented.

[0042] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0043] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the steps in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0044] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0045] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0046] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in the present invention should be included in the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims.

Claims

1. A method for implementing cross-network data synchronization using a scheduling algorithm based on an isolation device, characterized by: The steps include: Step 1: Configure the external database, enable archive logs to capture changes to table objects, and develop a log parsing service to parse archive logs into standard SQL statements. Step 2: When users generate new structured and unstructured data in the external network operating system, the unstructured data is stored on the file server, and the file storage path is stored in the structured table. When the structured table data is changed, the archive log is parsed to generate SQL statements and written into the external network structured data record table T_DATA_W. The file storage path is recorded using the external network unstructured path record table T_FILE_W, and structured and unstructured records are stored and processed separately. Step 3: Deploy scheduling service program A on the external network server and read the data recorded in the external network structured data record table T_DATA_W. The external network structured data record table T_DATA_W records the SQL statements for structured data changes. Leveraging the characteristics of the isolation device, the SQL statements are directly executed in the internal network database through the isolation device proxy to achieve structured data synchronization. Step 4: Deploy the scheduling service program B on the external network server, read the file storage path recorded in the external network unstructured path record table T_FILE_W, read the file path to generate a file stream and write it into the SQL statement variable, execute the SQL statement storing the file stream information directly in the internal network database through the change record isolation device agent, insert it into the internal network unstructured file stream information record table T_FILE_N for storage, and realize cross-region synchronization of file data; Step 5: Deploy the scheduling service program C on the intranet server, read the records in the intranet unstructured file stream information record table T_FILE_N, and generate a local temporary unstructured file from the file stream in the scheduling service program. Then, upload the file data to the intranet file server, and finally complete the file synchronization. Step 6: After the synchronization of structured data and unstructured data is completed, delete the synchronized records in the T_DATA_W, T_FILE_W, and T_FILE_N tables, and delete the temporary unstructured files to release resources.

2. The method for implementing cross-network data synchronization using a scheduling algorithm based on an isolation device according to claim 1, characterized in that: The isolation device is arranged between the information intranet and the information extranet, and is used to achieve high-intensity logical isolation between the information intranet and the information extranet systems and devices, and use the SQL proxy method to realize data exchange.

3. The method for implementing cross-network data synchronization using a scheduling algorithm based on an isolation device according to claim 1, characterized in that: The archive log is an inactive redo log backup of the database, and all redo history records are retained by using the archive log.

4. A system for realizing cross-network data synchronization based on an isolation device and a scheduling algorithm, characterized in that include: Computer-readable storage media and processor; The computer-readable storage medium is used to store executable instructions; The processor is used to read the executable instructions stored in the computer-readable storage medium and execute the method for implementing cross-network data synchronization based on the isolation device using a scheduling algorithm according to any one of claims 1-3.

5. A non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method for realizing cross-network data synchronization based on an isolation device using a scheduling algorithm as described in any one of claims 1 to 3.

Images