Cross-optical-shutter collection method and system for multi-data-source data

Abstract

The invention discloses a multi-data-source data cross-optical-gate collection method and system, and relates to a data security technology, and the method comprises the following steps: at a data access platform, notifying a data channel to start a task through a kafka message; receiving a kafka message, verifying the token of the received kafka message, and generating a data file after the token is verified; performing data compression and encryption on the generated data file, and generating a digital signature; performing data slicing on the data file after the digital signature is generated, and packaging each data slice based on a data proprietary protocol; and transmitting the packaged data packet by using a one-way network, writing audit data outside the optical gate, and crushing the data packet after writing. The embodiment of the invention provides a dual-channel verification method based on an account checking mechanism. On the basis of keeping the unidirectional transmission characteristic of the original data optical shutter, the data of the receiving end can be periodically verified through the independently deployed reverse optical shutter channel.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data security, in particular to a multi-data-source data cross-optical gate aggregation method and system. BACKGROUND

[0002] In today's digital era, data security transmission and cross-domain data integration have become the core issues of data security and informatization construction.

[0003] With the deepening of informatization construction, the demand for one-way secure transmission of data between classified networks and public networks is increasingly prominent. Especially in the context of smart city construction, e-government development and critical infrastructure protection, how to achieve efficient data aggregation while ensuring physical isolation has become a technical problem that needs to be solved in the current network security field.

[0004] Traditional data cross-domain transmission schemes mainly face three challenges: insufficient security, low efficiency and lack of flexibility. With the popularization of big data and cloud computing technologies, data volume grows exponentially, and traditional transmission methods have been unable to meet the real-time and high-throughput business needs.

[0005] Under this background, one-way optical gate technology emerged as the times require. Based on the one-way physical characteristics of light, it ensures that data can only flow from the low-security domain to the high-security domain in one direction, fundamentally eliminating the risk of high-class information leakage. However, existing optical gate systems rely on special interfaces and protocols (such as database synchronization, FTP), and have high coupling degree of software and hardware, making it difficult to adapt to the flexible needs of diversified business scenarios. SUMMARY

[0006] The embodiments of the present application provide a multi-data-source data cross-optical gate aggregation method and system, and propose a dual-channel verification method based on reconciliation mechanism. Specifically, while maintaining the original data optical gate one-way transmission characteristics, the receiving end data verification result is periodically verified through an independently deployed reverse optical gate channel.

[0007] The embodiments of the present application provide a multi-data-source data cross-optical gate aggregation method, which includes the following steps: In the data access platform, start the data channel task through the kafka message notification; Receive the kafka message, verify the token of the received kafka message, and generate a data file after verifying the token; Compress, encrypt and generate a digital signature for the generated data file; Slice the data file after generating the digital signature, and encapsulate each data slice based on a data-specific protocol; The encapsulated data packet is written into the audit data outside the optical gate by using one-way network transmission, and the data packet is crushed after being written.

[0008] The embodiment of the present application provides a multi-data-source-data cross-optical-gate collection system, comprising a processor and a memory, wherein the memory stores a computer program, and the computer program is executed by the processor to realize the steps of the multi-data-source-data cross-optical-gate collection method.

[0009] The embodiment of the present application provides a double-channel verification method based on a reconciliation mechanism. The data verification result of the receiving end can be periodically obtained through the independently deployed reverse optical gate channel on the basis of maintaining the original data optical gate one-way transmission characteristic.

[0010] The above description is only a summary of the technical scheme of the present application. In order to more clearly understand the technical means of the present application, the content of the specification can be implemented, and in order to make the above and other purposes, characteristics and advantages of the present application more obvious and easy to understand, the following specific embodiments of the present application are described. BRIEF DESCRIPTION OF DRAWINGS

[0011] Various other advantages and benefits will become apparent to those of ordinary skill in the art upon reading the following detailed description of the preferred embodiments. The accompanying drawings are included to provide a description of the preferred embodiments and are not meant to limit the present application. Moreover, the same reference numerals in the attached drawings indicate the same or similar components. In the drawings: Figure 1 The flowchart of the multi-data-source-data cross-optical-gate collection method of the embodiment of the present application is shown. DETAILED DESCRIPTION

[0012] Exemplary embodiments of the present disclosure will be described more fully hereinafter with reference to the accompanying drawings, in which exemplary embodiments of the present disclosure are shown. It is to be understood that the present disclosure can be embodied in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the present disclosure to those skilled in the art.

[0013] The embodiment of the present application provides a multi-data-source-data cross-optical-gate collection method, which integrates data of different data sources into a format required in the optical gate, compresses and encrypts the data, transmits the data in blocks, checks the integrity in the optical gate, imports the data into the database in the optical gate if the checking is successful, and transmits the data out through a second channel to inform the optical gate of the reason if the checking fails. Specifically, as shown in Figure 1 The method comprises the following steps: In step S101, the data access platform starts the data channel task through kafka message notification. In step S102, a kafka message is received, a token is verified for the received kafka message, and a data file is generated after the token is verified; In step S103, data compression, encryption, and digital signature generation are performed on the generated data file.

[0014] In step S104, the data file after the digital signature is generated is subjected to data slicing, and each data slice is encapsulated based on a data-specific protocol.

[0015] In step S105, the encapsulated data packet is written into the optical gate external audit data using one-way network transmission, and the data packet is pulverized and destroyed after being written.

[0016] Further comprising: In step S106, the data packet is received inside the optical gate, and the received data packet is reassembled; In step S107, the data signature is verified, and the integrity check is performed to import the data into the database inside the optical gate after the check passes.

[0017] In step S108, the second optical gate is used to feed back whether the data is successfully imported into the warehouse. In a specific example, the present application periodically returns the receiving end data check result (including but not limited to data packet serial number, CRC check value, timestamp, etc.) to the sending end in a secure manner through an independently deployed reverse optical gate channel (second optical gate).

[0018] The method of the present application constructs a closed-loop check system under the premise of strict physical isolation, adopts compression and encryption to ensure security, and transmits in blocks. For the packet that fails the integrity check, automatic retransmission is completed.

[0019] The method of the present application reduces the data transmission amount by compression and block division, and only needs to retransmit a certain block when a packet is lost. Compared with the previous performance, the transmission efficiency is improved by ten times. The present application has strong fault isolation and can maintain the accounting ability when the data flow is abnormal. In the test, even if the data packet loss rate reaches 1 / 1000, the accounting system is still 100% available.

[0020] In some embodiments, it further includes retransmitting only the any data packet in the case of loss of the any data packet.

[0021] In some embodiments, it further includes receiving a kafka message, and verifying a token for the received kafka message, comprising: Listening to the data message sent by the @KafkaListener, calling the handleMessage method to verify the message, and token verification; After verification, the readData method is called to call the corresponding implementation class according to the database type in the message to make the implementation class generate a unified file for the data of the corresponding database type. In a specific example, java is used for implementation, in the client, the data message sent by the data access is listened to through @KafkaListener, the handleMessage method is called to verify the message, token verification, after passing, the readData method is called to call the corresponding implementation class according to the database type in the message ftp: FtpFileReadImpl mysql: MysqlORCReadImpl dm: DmORCReadImpl clickhouse: ClickhouseReadImpl doris: DorisReadImpl Jupiter: JupiterReadImpl The corresponding implementation class will generate a unified file for the data of the corresponding database type, which is currently an orc file, to facilitate unified import into hive. After generating the file, the unified readData method is called, and the udp client is called for sending. The specific implementation of the client is as follows Client(UDPConnectionParam contentCore, FileCheckMapperfileCheckMapper, Queue <transferdictionaryentity>queue, TransferWorkflowMappertransferWorkflowMapper) { this.serverIP = contentCore.getUdpServerIp(); / / UDP configured IP this.port = contentCore.getUdpServerPort(); / / The UDP port configured this.callbackPort = contentCore.getUdpCallbackPort(); / / Port for sending back invoices this.publicKey = contentCore.getPublicKey(); / / Public key this.fileCheckMapper = fileCheckMapper; / / Audit log mapper this.queue = queue; / / Step queue this.transferWorkflowMapper = transferWorkflowMapper; this.pause = contentCore.getPause(); / / UDP transmission interval this.connectionParam = contentCore; / / Basic parameters }

[0022] In some embodiments, data compression and encryption of the generated data file includes: On the client side, the FileSplitUtil.createSplitZipFile method is called to compress the file; Use FileSm4Utils.encryptFile to encrypt the file.

[0023] In a specific example, the client will process the file by calling FileSplitUtil.createSplitZipFile. The method compresses the file, calls FileSm4Utils.encryptFile to encrypt the file, and generates... Then, based on the size, the data is divided into chunks, and each chunk is encapsulated into a fileEvent entity to send a UDP packet. A specific example is shown below: FileEvent() { private long totalFileSegments; / / Total number of segments private long fileSegmentNumber; / / Current segment number private long totalFileSize; / / Total size private long fileSegmentSize; / / Size of each segment private long segmentStartIndex; / / Start index private long segmentEndIndex; / / End index private byte[] fileSegmentData; / / File packet content @ByteSizeWithChartset(max = 1000) private String md5HashMapStr; / / MD5 value / / Code used as a unique identifier for reconciliation private String code; / / Unique identifier private intstatus; / / Status private String filename; / / File name private Long fileLine; / / Total number of lines in the file private Integer type; / / Type private String packetType; / / Database type private String fatherCode; / / Parent code private Integer retransferType; / / Retransfer type private Integer fileIndex; / / Child file marker private Integer fileCount; / / Size of child files private String childFilename; / / Name of child file private String datasourceGroup; / / Data source group } Adhering to the UDP protocol, UDP messages are sent via DatagramSocket.send(DatagramPacket) packets, and then audit logs are written to the database.

[0024] In some embodiments, transmitting the encapsulated data packet via a one-way network includes: Based on the size of the data packet, a fileEvent entity is encapsulated to send the UDP packet.

[0025] In some embodiments, receiving data packets within the light gate includes: On the server side, the ReceiveThread thread pool that has started to monitor the udp port is used to obtain the udp packet message sent by the client, and the ConcurrentLinkedQueue queue is used to sequentially receive the message; The received data packet is reorganized, including: After receiving, the WorkThread processing thread logic is started, the createAndWriteFile method is called to reorganize the data packet, and the file is written. In the case of the last data packet of the last file, the data warehousing method writeFile is triggered to reorganize all data packets.

[0026] Specifically, in the server, the ReceiveThread thread pool that has started to monitor the udp port is used to obtain the udp block message sent by the client, and the ConcurrentLinkedQueue queue is used to sequentially receive the message. After receiving, the WorkThread processing thread logic is started, the createAndWriteFile method is called to reorganize the data packet, and the file is written. In the case of the last data packet of the last file, the data warehousing method writeFile is triggered to reorganize all data packets, verify the digital signature, and perform integrity md5 check on the file. After verification, in the case of a large number of files, packet loss may occur, and an error may occur at the md5 step. If it is normal, the file is decrypted and decompressed, loaded into the database, and the audit date is stored in the database. The specific entity is as follows: FileCallbackEntity{ private Integer id; / / Primary key id private String code; / / Unique identifier private String fileName; / / File name private Long fileSize; / / File size private LocalDateTime receiveTime; / / Receive time private LocalDateTime callbackTime; / / Callback time private Long fileLine; / / Number of lines in the file private Integer fileType; / / File type (1 Structured, 2 Unstructured) private String successCode; / / s200 - Success s500 - Failure private String message; / / Message to be returned private String packetType; / / Source type private Integer receiveStatus; / / Receive status 0 - Failure 1 - Success private Integertype; / / Statement type (1 Data Channel Service, 2 Data Exchange Service, 3 Sample Library Transfer) private String errorReason; / / Error reason private String jobId; / / Task id private String jobName; / / Task name private LocalDateTime startTime; / / Start time private String fatherCode; / / Parent code private Integer retransferType; / / Retransfer type private String datasourceName; / / Data source name private Integer countIdentifier; / / Statistical identifier: 1-Full, 2-Incremental, used for cumulative statistics private String tableRemarks; / / Table name (in Chinese) private String datasourceGroup; / / Data source group } The second channel is used to transfer the data from the shutter to the outside of the shutter, indicating whether the data was successfully received or not. The data is then retransmitted, or the error message is fixed before retransmission.

[0027] The application innovatively designs a double-channel verification method based on a reconciliation mechanism, periodically returns the data verification results (including but not limited to data packet serial number, CRC check value, timestamp and the like information) of the receiving end to the sending end in a safe manner through an independently deployed reverse optical gate channel on the basis of maintaining the original data optical gate unidirectional transmission characteristics.

[0028] The scheme of the application realizes three technical breakthroughs: (1) constructing a closed-loop verification system under the premise of strictly maintaining physical isolation; (2) adopting compression encryption to realize block transmission under the premise of ensuring security; and (3) automatically completing retransmission for the packet that fails the integrity check.

[0029] The application embodiment further provides a multi-data-source data cross-optical gate collection system, including a processor and a memory, the memory stores a computer program, and the computer program is executed by the processor to realize the steps of the multi-data-source data cross-optical gate collection method as described above.

[0030] It should be noted that in the embodiments of the application, the terms "comprise", "contain" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or device. Without more limitations, the element defined by the statement "comprising a" does not exclude the presence of another identical element in the process, method, article or device including the element.

[0031] The above application embodiment serial numbers are only for description, not representing the advantages and disadvantages of the embodiments.

[0032] Through the description of the above embodiments, those skilled in the art can clearly understand that the above-mentioned embodiment method can be realized by means of software and necessary general hardware platform, of course, it can also be realized by hardware, but in many cases, the former is a better embodiment. Based on such understanding, the technical solutions of the application can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a plurality of instructions for making a terminal (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) execute the method described in each embodiment of the application.

[0033] The embodiments of the present application are described above with reference to the accompanying drawings, but the present application is not limited to the specific embodiments described above, and the specific embodiments described above are merely illustrative, but not restrictive, and a person of ordinary skill in the art can make many forms under the inspiration of the present application without departing from the purpose of the present application and the scope protected by the claims, and these all belong to the protection of the present application.< / transferdictionaryentity>

Claims

1. A method for cross-optical gate aggregation of data from multiple data sources, characterized in that, Includes the following steps: On the data access platform, the data channel task is started via Kafka message notification. Receive Kafka messages, verify the token of the received Kafka messages, and generate data files after verifying the token; The generated data file is compressed, encrypted, and a digital signature is generated. The digitally signed data file will be sliced, and each data slice will be encapsulated based on a proprietary data protocol. The encapsulated data packet is transmitted via a one-way network and written into external audit data of the optical gate. After writing, the data packet is shredded.

2. The method for cross-optical gate aggregation of multi-source data as described in claim 1, characterized in that, Also includes: Data packets are received inside the optical shutter, and the received data packets are reassembled. Verify the data signature and perform integrity checks so that the data can be imported into the database within the light gate after the checks pass. The second optical shutter is used to check whether the data outside the optical shutter has been successfully entered into the database.

3. The method for cross-optical gate data aggregation from multiple data sources as described in claim 2, characterized in that, It also includes retransmitting only the lost data packet.

4. The method for cross-optical gate aggregation of multi-source data as described in claim 2, characterized in that, It also includes receiving Kafka messages and verifying the token for the received Kafka messages, including: Listen for data messages sent by the data inlet using @KafkaListener, and call the handleMessage method to verify the message and token. After successful verification, the readData method is called to invoke the corresponding implementation class based on the database type in the message, so that the implementation class can generate a unified file for the corresponding database type.

5. The method for cross-optical gate aggregation of multi-source data as described in claim 4, characterized in that, Data compression and encryption of the generated data files include: On the client side, the FileSplitUtil.createSplitZipFile method is called to compress the file; Use FileSm4Utils.encryptFile to encrypt the file.

6. The method for cross-optical gate aggregation of multi-source data as described in claim 4, characterized in that, The encapsulated data packet is then transmitted via a one-way network, including: Based on the size of the data packet, a fileEvent entity is encapsulated to send the UDP packet.

7. The method for cross-optical gate aggregation of multi-source data as described in claim 6, characterized in that, Receiving data packets within the light gate includes: On the server side, UDP packet messages sent by the client are obtained through the ReceiveThread thread pool that has been started listening on the UDP port, and are received sequentially through the ConcurrentLinkedQueue queue; Reassembling received data packets includes: Upon receiving the data, the WorkThread processing thread logic is started, the createAndWriteFile method is called to reassemble the data packets and write them to the file. If the current data packet is the last data packet in the last file, the data entry task is triggered, the writeFile method is called to reassemble all the data packets.

8. A cross-optical gate data collection system for multi-source data, characterized in that, It includes a processor and a memory, wherein the memory stores a computer program, which, when executed by the processor, implements the steps of the cross-gate aggregation method for multi-source data as described in any one of claims 1 to 7.

Images