Cross-border data transmission system based on multi-source heterogeneous data encryption transmission

Abstract

The invention discloses a cross-border data transmission system based on multi-source heterogeneous data encryption transmission. The cross-border data transmission system comprises an overseas data forwarding module, a cloud transmission module and a domestic data circulation module, the data forwarding module comprises a data grading processing module and a data synchronization module; the data grading processing module is used for carrying out data grading, data scheduling, data encryption and data compression on overseas data; the data synchronization module executes a data synchronization application program; the cloud transmission module comprises a server-side framework, a data synchronization algorithm and an encryption transmission module, and is used for receiving overseas data transmitted by the data synchronization module and transmitting the overseas data to the domestic data circulation module; the domestic data circulation module is used for data receiving, data decompression and data decryption; the problem that in the prior art, data transmission in power enterprise business data cross-border management cannot meet the cross-border data safety management requirement is solved.

Description

Technical Field

[0001] The present invention relates to the technical field of cross-border data transmission, and in particular to a cross-border data transmission system based on encrypted transmission of multi-source heterogeneous data. Background Art

[0002] At present, the energy business of large domestic power companies includes a large number of overseas businesses, including the acquisition of some overseas companies to undertake the distribution and sale of electricity, secondary transmission and power generation business of overseas companies; a large number of overseas businesses are affected by their own regional differences, business type differences, development status differences, regulatory differences and technical path differences, which makes it very inconvenient to manage their business-related data; under the existing technology, some overseas business data are still managed through their own independent management systems. When the domestic corporate headquarters needs to aggregate and process data, they need dedicated personnel to collect data and process the corresponding data, so as to integrate the required data with the comprehensive management platform of the current domestic enterprises. In the specific operation process, there are many projects, chaotic management and low efficiency; therefore, it is considered to manage data by building a comprehensive management platform involving cross-border data management in the electrical industry. Data transmission, as a key channel to achieve unified domestic management of overseas data, needs to meet the following requirements: Cross-border data compression requirements: reduce the cost and pressure of cross-border data transmission through reasonable compression methods while ensuring performance; cross-border data encryption requirements: formulate a full-process encryption strategy to ensure that data from file transfer to storage is encrypted during the transmission process to ensure data security; cross-border instant data synchronization requirements: synchronize various changes in real time, including historical data, real-time data, new data and reported data of the data source to meet the company's requirements for timeliness; multi-role permission management requirements: realize flexible data permission management, set specific permissions according to the needs of different personnel, so as to facilitate management and control; data classification management requirements: flexibly set security levels for data of different contents, and data of different levels can enjoy Subject to different levels of transmission services, high-level data requires higher permissions to prevent misoperation of critical information; security service supervision: security service supervision of personnel with different permissions and roles, combined with authentication functions such as mobile phone text messages, to ensure safe and reliable operations during data transmission; international transmission multi-channel node backup: to ensure the stability and continuity of the transmission channel, adopt multi-node backup capabilities, and when a channel fails, the backup channel can be used to maintain data transmission; multi-terminal transmission sharing: support mobile devices, data and information synchronization and sharing between clients and servers, simplify transmission and sharing operations between different devices; unified transmission protocol: use a unified transmission protocol to transmit data, ensure the standardization of encryption and parsing, and reduce workload.

[0003] The existing cross-border data transmission technology has not yet provided a solution to the above technical requirements. Therefore, it is necessary to design a cross-border data transmission system based on multi-source heterogeneous data encrypted transmission to solve the above technical problems. Summary of the invention

[0004] The technical problem to be solved by the present invention is to provide a cross-border data transmission system based on multi-source heterogeneous data encryption transmission, so as to solve the problem in the prior art that data transmission in the cross-border management of business data of power enterprises cannot meet the cross-border data security management requirements. The overseas business data can be forwarded to the cloud server in an orderly, complete and efficient manner, and connected to the cross-border data transmission channel to ensure that specific data can circulate in a cycle, which can meet the data security management requirements of overseas and the company.

[0005] In order to solve the above technical problems, the technical solution adopted by the present invention is: A cross-border data transmission system based on multi-source heterogeneous data encrypted transmission, including overseas data forwarding module, cloud transmission module and domestic data circulation module; The data forwarding module includes a data classification processing module and a data synchronization module; the data classification processing module performs data classification, data scheduling, data encryption and data compression on overseas data; the data synchronization module executes the data synchronization application program and transmits overseas data to the cloud transmission module in the form of quasi-real-time scheduled tasks; The cloud transmission module includes a server-side architecture, a data synchronization algorithm, and an encryption transmission module, which receives overseas data transmitted by the data synchronization module and transmits the overseas data to the domestic data circulation module; The domestic data flow module is used for data reception, data decompression and data decryption, and transmits the data transmitted by the cloud transmission module from network zone 4 to network zone 3, realizing the conversion of data from the external network to the internal network.

[0006] Preferably, data classification includes dividing overseas data into dynamic data, static data and time series data; data scheduling includes text processing and building scheduling tasks, and text processing is used to read structured text data and unstructured text data in the data after data classification, and build scheduling tasks for the text-processed data.

[0007] Preferably, the data encryption adopts a symmetric encryption algorithm, and the data is encrypted by using a plain text plus a secret key pair. The specific method is: The input 64-bit data block is reassembled bit by bit, and the output is divided into two parts, L0 and R0, each part is 32 bits long; Set the permutation rule, and replace the 58th bit of the input with the first bit, the 50th bit with the second bit, and so on. The last bit is the original 7th bit. L0 and R0 are the two parts of the output after the permutation. L0 is the left 32 bits of the output, and R0 is the right 32 bits. The EBC mode is used for grouping, including multiple plaintext blocks connected to the encryptor, and the plaintext blocks correspond to the output ciphertext blocks; the encryptor is connected to the secret key.

[0008] Preferably, data compression is performed by preprocessing compression, online compression, data block compression and incremental compression. The compressed data enters the collection and storage database. The terminal user who has passed the device password protection verification transmits the data to the collection and storage database through data sharing. The data compression specifically includes: Preprocessing compression: data preprocessing compression is performed through data encoding; Online compression, data is compressed online through LZ0\LZ4; Data block compression, data block compression is performed through bzip2\zip; Incremental compression, incremental compression is performed through Zlib\gzip.

[0009] Preferably, the data synchronization application includes batch data synchronization tasks, periodic incremental data synchronization tasks, time series data synchronization tasks and data synchronization status check tasks; the data synchronization application synchronizes the data in the collection and storage database to the cloud transmission module according to the data update cycle by means of scheduled tasks, and the specific execution process is as follows: Batch data synchronization tasks: The full data acquisition API is called through task configuration to obtain the full data from the collection and storage database, map and parse the data, and organize it into a text file with a tree structure. The file content is a text field of structured data encrypted by code; the text file is synchronized by calling the interface of the cloud transmission module; Periodic incremental data synchronization task: The task configuration is used to trigger the call of the periodic incremental data synchronization task. Different types of data incremental acquisition APIs are called according to different data update cycles to obtain corresponding types of data from the collection and storage database. The interface logic is used to calculate the difference update data set to obtain incremental data, and the text file is synchronized by calling the interface of the cloud transmission module. Real-time data synchronization tasks: Scan and monitor data updates in real time through task configuration. If there is data update, trigger the real-time data acquisition API to obtain real-time data from the collection and storage database; obtain and update real-time data through the patrol mechanism; synchronize text files by calling the interface of the cloud transmission module; Data synchronization status check task: Obtain data synchronization status from the collection storage database through the data synchronization status acquisition API; Check the data synchronized in the cloud transmission module through the data synchronization status check API; After the batch data synchronization task, periodic incremental data synchronization task, and real-time data synchronization task are completed, the result status of data synchronization is checked. If it fails, synchronization is re-triggered. If the task execution still fails after repeated attempts, a message notification is triggered to manually troubleshoot and solve the data synchronization problem.

[0010] Preferably, the server-side architecture of the cloud transmission module is a distributed storage architecture, which stores user data in multiple nodes in a dispersed manner.

[0011] Preferably, the data synchronization algorithm of the cloud transmission module is a block-based incremental synchronization algorithm, which divides the file into blocks and compares the block differences between different nodes, and only transmits different blocks, thereby reducing the amount of data transmitted.

[0012] Preferably, the encrypted transmission module of the cloud transmission module includes transmission cloud encryption and storage encryption. The transmission cloud encryption adopts SSL / TLS encryption, and the storage encryption adopts AES256 encryption. AES256 encryption divides the file into many parts of 4MB each, and stores the blocks on different server hard disks for physical isolation.

[0013] Preferably, the encryption transmission module of the cloud transmission module includes a security service supervision module, and the security service supervision module includes: Firewalls, used to block attacks and prohibit access by unauthorized users and devices; Authentication server, used for password authentication and dynamic identity authentication; Application logs are used to record behavior. By recording user behavior, audit functions and unlimited historical version recovery functions can be implemented; Permission management, used for multi-role permission management, administrators control access rights.

[0014] Preferably, in the domestic data flow module, overseas data is sent to the cloud transmission module, and the transfer program downloads the data file to the four-zone server and parses it by connecting to the cloud transmission module API interface, and then sends the data to the big data platform of the intranet.

[0015] The beneficial effects of the cross-border data transmission system based on multi-source heterogeneous data encrypted transmission provided by the present invention are as follows: The present invention solves the problem in the prior art that data transmission in the cross-border management of business data of power enterprises cannot meet the requirements of cross-border data security management. It can forward overseas business data to the cloud server in an orderly, complete and efficient manner, and connect it to the cross-border data transmission channel to ensure that specific data can circulate in a cycle, thereby meeting the data security management requirements of overseas and the company. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The present invention will be further described below in conjunction with the accompanying drawings and embodiments: Figure 1 is a schematic diagram of the system architecture of the present invention; Figure 2 It is a schematic diagram of the architecture of the data forwarding module in the present invention; Figure 3 It is a flow chart of the data forwarding module in the present invention; Figure 4 It is a flow chart of the data synchronization module in the present invention; Figure 5 It is a flow chart of the cloud transmission module in the present invention. DETAILED DESCRIPTION

[0017] like Figure 1~Figure 5 As shown, a cross-border data transmission system based on multi-source heterogeneous data encryption transmission includes an overseas data forwarding module, a cloud transmission module and a domestic data circulation module; The data forwarding module includes a data classification processing module and a data synchronization module; the data classification processing module performs data classification, data scheduling, data encryption and data compression on overseas data; the data synchronization module executes the data synchronization application program and transmits overseas data to the cloud transmission module in the form of quasi-real-time scheduled tasks; The cloud transmission module includes a server-side architecture, a data synchronization algorithm, and an encryption transmission module, which receives overseas data transmitted by the data synchronization module and transmits the overseas data to the domestic data circulation module; The domestic data flow module is used for data reception, data decompression and data decryption, and transmits the data transmitted by the cloud transmission module from network zone 4 to network zone 3, realizing the conversion of data from the external network to the internal network.

[0018] Preferably, data classification includes dividing overseas data into dynamic data, static data and time series data; data scheduling includes text processing and building scheduling tasks, and text processing is used to read structured text data and unstructured text data in the data after data classification, and build scheduling tasks for the text-processed data.

[0019] Preferably, the data encryption adopts a symmetric encryption algorithm, and the data is encrypted by using a plain text plus a secret key pair. The specific method is: The input 64-bit data block is reassembled bit by bit, and the output is divided into two parts, L0 and R0, each part is 32 bits long; Set the permutation rule, and replace the 58th bit of the input with the first bit, the 50th bit with the second bit, and so on. The last bit is the original 7th bit. L0 and R0 are the two parts of the output after the permutation. L0 is the left 32 bits of the output, and R0 is the right 32 bits. The EBC mode is used for grouping, including multiple plaintext blocks connected to the encryptor, and the plaintext blocks correspond to the output ciphertext blocks; the encryptor is connected to the secret key.

[0020] Preferably, data compression is performed by preprocessing compression, online compression, data block compression and incremental compression. The compressed data enters the collection and storage database. The terminal user who has passed the device password protection verification transmits the data to the collection and storage database through data sharing. The data compression specifically includes: Preprocessing compression: data preprocessing compression is performed through data encoding; Online compression, data is compressed online through LZ0\LZ4; Data block compression, data block compression is performed through bzip2\zip; Incremental compression, incremental compression is performed through Zlib\gzip.

[0021] Preferably, the data synchronization application includes batch data synchronization tasks, periodic incremental data synchronization tasks, time series data synchronization tasks and data synchronization status check tasks; the data synchronization application synchronizes the data in the collection and storage database to the cloud transmission module according to the data update cycle by means of scheduled tasks, and the specific execution process is as follows: Batch data synchronization tasks: The full data acquisition API is called through task configuration to obtain the full data from the collection and storage database, map and parse the data, and organize it into a text file with a tree structure. The file content is a text field of structured data encrypted by code; the text file is synchronized by calling the interface of the cloud transmission module; Periodic incremental data synchronization task: The task configuration is used to trigger the call of the periodic incremental data synchronization task. Different types of data incremental acquisition APIs are called according to different data update cycles to obtain corresponding types of data from the collection and storage database. The interface logic is used to calculate the difference update data set to obtain incremental data, and the text file is synchronized by calling the interface of the cloud transmission module. Real-time data synchronization tasks: Scan and monitor data updates in real time through task configuration. If there is data update, trigger the real-time data acquisition API to obtain real-time data from the collection and storage database; obtain and update real-time data through the patrol mechanism; synchronize text files by calling the interface of the cloud transmission module; Data synchronization status check task: Obtain data synchronization status from the collection storage database through the data synchronization status acquisition API; Check the data synchronized in the cloud transmission module through the data synchronization status check API; After the batch data synchronization task, periodic incremental data synchronization task, and real-time data synchronization task are completed, the result status of data synchronization is checked. If it fails, synchronization is re-triggered. If the task execution still fails after repeated attempts, a message notification is triggered to manually troubleshoot and solve the data synchronization problem.

[0022] Preferably, the server-side architecture of the cloud transmission module is a distributed storage architecture, which stores user data in multiple nodes in a dispersed manner.

[0023] Preferably, the data synchronization algorithm of the cloud transmission module is a block-based incremental synchronization algorithm, which divides the file into blocks and compares the block differences between different nodes, and only transmits different blocks, thereby reducing the amount of data transmitted.

[0024] Preferably, the encrypted transmission module of the cloud transmission module includes transmission cloud encryption and storage encryption. The transmission cloud encryption adopts SSL / TLS encryption, and the storage encryption adopts AES256 encryption. AES256 encryption divides the file into many parts of 4MB each, and stores the blocks on different server hard disks for physical isolation.

[0025] Preferably, the encryption transmission module of the cloud transmission module includes a security service supervision module, and the security service supervision module includes: Firewalls, used to block attacks and prohibit access by unauthorized users and devices; Authentication server, used for password authentication and dynamic identity authentication; Application logs are used to record behavior. By recording user behavior, audit functions and unlimited historical version recovery functions can be implemented; Permission management, used for multi-role permission management, administrators control access rights.

[0026] Preferably, in the domestic data flow module, overseas data is sent to the cloud transmission module, and the transfer program downloads the data file to the four-zone server and parses it by connecting to the cloud transmission module API interface, and then sends the data to the big data platform of the intranet.

Claims

1. A cross-border data transmission system based on encrypted transmission of multi-source heterogeneous data, characterized in that: It includes overseas data forwarding module, cloud transmission module and domestic data circulation module; The data forwarding module includes a data classification processing module and a data synchronization module; the data classification processing module performs data classification, data scheduling, data encryption and data compression on overseas data; the data synchronization module executes the data synchronization application program and transmits overseas data to the cloud transmission module in the form of quasi-real-time scheduled tasks; The cloud transmission module includes a server-side architecture, a data synchronization algorithm, and an encryption transmission module, which receives overseas data transmitted by the data synchronization module and transmits the overseas data to the domestic data circulation module; The domestic data flow module is used for data reception, data decompression and data decryption, and transmits the data transmitted by the cloud transmission module from network zone 4 to network zone 3, realizing the conversion of data from the external network to the internal network.

2. The cross-border data transmission system based on multi-source heterogeneous data encrypted transmission according to claim 1 is characterized in that: Data classification includes dividing overseas data into dynamic data, static data and time series data; data scheduling includes text processing and building scheduling tasks. Text processing is used to read structured text data and unstructured text data in the data after data classification, and build scheduling tasks for the data after text processing.

3. The cross-border data transmission system based on multi-source heterogeneous data encrypted transmission according to claim 2 is characterized in that: Data encryption uses a symmetric encryption algorithm and encrypts data using plain text plus a secret key pair. The specific method is: The input 64-bit data block is reassembled bit by bit, and the output is divided into two parts, L0 and R0, each part is 32 bits long; Set the permutation rule, and replace the 58th bit of the input with the first bit, the 50th bit with the second bit, and so on. The last bit is the original 7th bit. L0 and R0 are the two parts of the output after the permutation. L0 is the left 32 bits of the output, and R0 is the right 32 bits. The EBC mode is used for grouping, including multiple plaintext blocks connected to the encryptor, and the plaintext blocks correspond to the output ciphertext blocks; the encryptor is connected to the secret key.

4. The cross-border data transmission system based on multi-source heterogeneous data encrypted transmission according to claim 1 is characterized in that: Data compression compresses data through pre-processing compression, online compression, data block compression and incremental compression. The compressed data enters the collection and storage database. The terminal user who has passed the device password protection verification transmits the data to the collection and storage database through data sharing; Data compression specifically includes: Preprocessing compression: data preprocessing compression is performed through data encoding; Online compression, data is compressed online through LZ0\LZ4; Data block compression, data block compression is performed through bzip2\zip; Incremental compression, incremental compression is performed through Zlib\gzip.

5. The cross-border data transmission system based on multi-source heterogeneous data encrypted transmission according to claim 1 is characterized in that: The data synchronization application includes batch data synchronization tasks, periodic incremental data synchronization tasks, time series data synchronization tasks, and data synchronization status check tasks. The data synchronization application synchronizes the data in the collection and storage database to the cloud transmission module according to the data update cycle through the scheduled task method. The specific execution process is as follows: Batch data synchronization tasks: The full data acquisition API is called through task configuration to obtain the full data from the collection and storage database, map and parse the data, and organize it into a text file with a tree structure. The file content is a text field of structured data encrypted by code; the text file is synchronized by calling the interface of the cloud transmission module; Periodic incremental data synchronization task: The task configuration is used to trigger the call of the periodic incremental data synchronization task. Different types of data incremental acquisition APIs are called according to different data update cycles to obtain corresponding types of data from the collection and storage database. The interface logic is used to calculate the difference update data set to obtain incremental data, and the text file is synchronized by calling the interface of the cloud transmission module. Real-time data synchronization tasks: Scan and monitor data updates in real time through task configuration. If there is data update, trigger the real-time data acquisition API to obtain real-time data from the collection and storage database; obtain and update real-time data through the patrol mechanism; synchronize text files by calling the interface of the cloud transmission module; Data synchronization status check task: Obtain data synchronization status from the collection storage database through the data synchronization status acquisition API; Check the data synchronized in the cloud transmission module through the data synchronization status check API; After the batch data synchronization task, periodic incremental data synchronization task, and real-time data synchronization task are completed, the result status of data synchronization is checked. If it fails, synchronization is re-triggered. If the task execution still fails after repeated attempts, a message notification is triggered to manually troubleshoot and solve the data synchronization problem.

6. The cross-border data transmission system based on multi-source heterogeneous data encrypted transmission according to claim 1 is characterized in that: The server-side architecture of the cloud transmission module is a distributed storage architecture, which stores user data in multiple nodes.

7. The cross-border data transmission system based on multi-source heterogeneous data encryption transmission according to claim 6 is characterized in that: The data synchronization algorithm of the cloud transmission module is a block-based incremental synchronization algorithm. By dividing the files into blocks and comparing the block differences between different nodes, only different blocks are transmitted to reduce the amount of data transmitted.

8. The cross-border data transmission system based on multi-source heterogeneous data encrypted transmission according to claim 7 is characterized in that: The encryption transmission module of the cloud transmission module includes transmission cloud encryption and storage encryption. The transmission cloud encryption adopts SSL / TLS encryption, and the storage encryption adopts AES256 encryption. AES256 encryption divides the file into many parts of 4MB each, and stores them in blocks on different server hard disks for physical isolation.

9. The cross-border data transmission system based on multi-source heterogeneous data encrypted transmission according to claim 8 is characterized in that: The encrypted transmission module of the cloud transmission module includes a security service supervision module, and the security service supervision module includes: Firewalls, used to block attacks and prohibit access by unauthorized users and devices; Authentication server, used for password authentication and dynamic identity authentication; Application logs are used to record behavior. By recording user behavior, audit functions and unlimited historical version recovery functions can be implemented; Permission management, used for multi-role permission management, administrators control access rights.

10. The cross-border data transmission system based on multi-source heterogeneous data encryption transmission according to claim 1 is characterized in that: In the domestic data flow module, overseas data is sent to the cloud transmission module. The transfer program downloads the data file to the four-zone server and parses it by connecting to the cloud transmission module API interface, and then sends the data to the big data platform of the intranet.

Images