Gap-based digital detection data transmission method and system

Abstract

The invention provides a digital detection data transmission method and system based on a gatekeeper, and belongs to the technical field of computer software, and the method comprises the steps: S1, defining basic parameters; s2, defining a data model, and configuring and managing model attribute information needing to be driven at two ends through a model management page; s3, starting transaction flow scheduling, and capturing a to-be-transmitted data flow by a transaction method; performing unified processing on data generated in each library table in a scattered manner in different time periods in an asynchronous manner; s4, encrypting and packaging the grabbed data, initiating transmission of a to-be-transmitted data packet at an industrial control network end by calling a one-way gatekeeper interface of an enterprise, and receiving the data at a secret-related network end by calling a service interface through the one-way gatekeeper; and S5, performing data distribution in a synchronous mode according to the original time sequence during data capture. According to the method, various complex, instant and high-compliance cross-network-segment data transmission requirements can be met.

Description

Technical Field

[0001] The invention belongs to the technical field of computer software, and in particular relates to a digital detection data transmission method and system based on a network gate. Background Art

[0002] With the rapid development of intelligent manufacturing, the demand for data interaction between confidential campus networks and industrial control networks is increasing. Traditional data interaction methods have problems of low efficiency and insufficient security. At present, based on the need for cross-network transmission, some military enterprises have carried out pilot projects of single-item network gates, which can realize data transmission through network gates. Since digital detection systems are usually deployed in industrial control networks, a large amount of collected data needs to be transmitted to confidential campus networks for analysis. How to achieve efficient cross-network transmission has become an important issue currently faced. At present, the data transmission between the two networks mainly includes the following solutions: 1. Capture structured SQL records through system logs and package and transmit the records; 2. Based on the data export tool provided by MySQL, the incremental data change log is captured. However, the above solutions have limited support for the transmission of large files and data transmission of DM databases under the situation of localization. Therefore, how to deal with large attachments, adapt to DM databases, and transmit data across networks under the condition of one-way network gates has become an urgent problem to be solved. Summary of the invention

[0003] In order to solve the above technical problems, the present invention provides a digital detection data transmission method based on a network gate, comprising the following steps:

[0004] Step S1: define basic parameters, including: data interface address, data interface configuration rules, file access path, and various parameters related to data flow transaction scheduling;

[0005] Step S2: define the data model, and configure and manage the model attribute information that needs to be driven at both ends through the model management page, including: execution sequence, model name, model content description, model keywords, and corresponding buffer model;

[0006] Step S3: Start transaction flow scheduling, and capture the transmitted data flow in a transactional manner; use an asynchronous method to uniformly process the data generated in different time periods in various database tables, while ensuring that the original time sequence of the data can be retained after being captured;

[0007] Step S4: Encrypt and package the captured data, initiate transmission of the data packet to be transmitted by calling the enterprise's one-way network gate interface on the industrial control network side, and receive data by calling the service interface through the one-way network gate on the confidential network side;

[0008] Step S5: Distribute data in a synchronous manner according to the original timing when the data is captured; introduce distributed data distribution to meet the synchronization requirements of large-volume data.

[0009] Beneficial effects:

[0010] The present invention provides a digital detection data transmission method and system based on a network gate, which can effectively improve the security and efficiency of data interaction, support parallel processing mechanism, can process multiple data interaction tasks at the same time, improve overall processing efficiency, and meet the high requirements of the intelligent manufacturing field for data synchronization and transmission; support multiple databases, including Oracle, MySQL, SQL Server, and domestic DAMO database, to meet the diverse data storage needs in the intelligent manufacturing field, and solve various complex, instant, and highly compliant cross-segment data transmission needs. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 A schematic flow chart of a digital detection data transmission method based on a network gate according to the present invention;

[0012] Figure 2 This is an example diagram of an application scenario of the method of the present invention;

[0013] Figure 3 The present invention is a structural block diagram of a digital detection data transmission system based on a network gate. DETAILED DESCRIPTION

[0014] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0015] Embodiment 1

[0016] like Figure 1 As shown, a digital detection data transmission method based on a network gate provided by an embodiment of the present invention includes the following steps:

[0017] Step S1: define basic parameters, including: data interface address, data interface configuration rules, file access path, and various parameters related to data flow transaction scheduling;

[0018] Step S2: define the data model, and configure and manage the model attribute information that needs to be driven at both ends through the model management page, including: execution sequence, model name, model content description, model keywords, and corresponding buffer model;

[0019] Step S3: Start transaction flow scheduling, and capture the transmitted data flow in a transactional manner; use an asynchronous method to uniformly process the data generated in different time periods in various database tables, while ensuring that the original time sequence of the data can be retained after being captured;

[0020] Step S4: Encrypt and package the captured data, initiate transmission of the data packet to be transmitted by calling the enterprise's one-way network gate interface on the industrial control network side, and receive data by calling the service interface through the one-way network gate on the confidential network side;

[0021] Step S5: Distribute data in a synchronous manner according to the original timing when the data is captured; introduce distributed data distribution to meet the synchronization requirements of large-volume data.

[0022] In one embodiment, the above step S1: defining basic parameters includes: data interface address, data interface configuration rules, file access path and various parameters related to data flow transaction scheduling, specifically including:

[0023] Define the data source information to determine the data interface address, data interface configuration rules, file access path and various parameters related to data flow transaction scheduling related to data transmission. The present invention supports data transmission of various types of databases, such as Oracle, MySQL, SQL Server, and domestic DAMO database.

[0024] In one embodiment, the above step S2: defines the data model, and configures and manages the model attribute information that needs to be driven at both ends through the model management page, including: execution sequence, model name, model content description, model keyword, corresponding buffer model, specifically including:

[0025] Define the basic properties of the digital model. For example, library table maintenance includes: library table information, including execution sequence, entity name, entity description, primary key, and buffer table name. Configure the list of libraries and tables for data exchange at the data sending end, and perform initialization work based on this to generate the corresponding data synchronization model. The method of the present invention supports real-time changes to the data model.

[0026] In one embodiment, the above step S3: starting transaction flow scheduling, capturing the data flow to be transmitted in a transactional manner; using an asynchronous method to uniformly process the data generated in different time periods in various database tables, while ensuring that the original time sequence of the data can be retained after being captured, specifically includes:

[0027] After the maintenance of the library table information in step S2 is completed, the parameters of the scheduler are configured according to the operating environment of the system (such as the production environment, the test environment), the data stream is captured by the transaction method, and the captured data is stored in the cache table, which usually involves executing INSERT statements or batch insert operations. Then the data generated in different time periods in various library tables are processed uniformly. To improve efficiency, the present invention uses an asynchronous method to process the data, while ensuring that the data can still retain its original time sequence after being captured. In addition, the present invention identifies and processes large fields in the database, generates binary files, and establishes a mapping relationship with the corresponding data rows.

[0028] In one embodiment, the above step S4: encrypting and packaging the captured data, initiating transmission of the data packet to be transmitted by calling the one-way network gate interface of the enterprise on the industrial control network side, and receiving the data by calling the service interface through the one-way network gate on the confidential network side, specifically includes:

[0029] During the data packaging process, select appropriate encryption algorithms (such as AES) and signature algorithms (such as RSA, ECDSA) for the content according to security requirements and performance requirements to ensure that the data is transmitted in ciphertext and cannot be tampered with. After the packaging task is completed, the data table is uploaded to the specified path and waits for the network gate receiving end to capture it. After the data is sent, the initialization result of the list is automatically updated. At the same time, the present invention also supports manual execution of transmission tasks. The transmission service can be stopped by [Close Data Transmission Task].

[0030] When abnormal transmission behavior is detected, the abnormal capture mechanism is immediately triggered to capture the specific information of the problem, and then the captured abnormal information, related data packets, timestamps, transmission logs and other key data are packaged into a complete fault report. The packaged fault report is safely stored on a reliable storage medium to ensure the integrity and accessibility of the data. After storing enough data, professional data analysis tools can be used to compare the data between the two networks to find data differences and problems in the transmission process. According to the results of the data comparison, combined with the transmission log and abnormal information, the specific reasons for the data differences and transmission problems are analyzed.

[0031] The present invention also backs up the transmitted data and classifies and stores the archived data according to the data type, time or business value, which helps to quickly locate and retrieve the required data. Indexes and metadata are created for the archived data to quickly search and locate specific data.

[0032] In one embodiment, the above step S5: distributing data in a synchronous manner according to the original timing when the data is captured; using distributed data distribution for large-volume data synchronization requirements, specifically includes:

[0033] 1. Data file synchronization: First, clarify the data model, which usually includes defining the table structure (such as field types, constraints, etc.), relationships (such as foreign keys), and any indexes and views in the database to ensure that the data model can clearly reflect business needs. Then create the table structure in the database and set constraints such as primary keys and foreign keys as needed; next, create triggers for the data tables that need to be synchronized, and generate corresponding SQL files through table triggers.

[0034] 2. Text file synchronization: Especially when it comes to large fields stored in the database (such as BLOB or TEXT type fields, which are often used to store large amounts of text or binary data), a series of steps can be used to achieve data reading, encryption, naming, and the final file storage and association with database records. Before writing large field data to a file, encryption is required to ensure data security, which can be achieved by using encryption algorithms such as AES. For each record, the file can be named based on its unique identifier in the database (such as ID), and the file name can be stored in an additional field in the database. Finally, the encrypted data is written to a file named after the database record ID.

[0035] 3. Streaming media and image file synchronization: On the sending end, you can use the file system monitoring API provided by the operating system (such as Windows' FileSystemWatcher and Linux's inotify) to monitor changes in the specified folder. Set the scanning frequency according to the needs, which can be a scheduled scan or a real-time trigger when there is a change in the folder. According to security and performance requirements, use ready-made libraries (such as Python's cryptography library) to perform encryption operations, ensuring that a unique key is generated for each file or using the same key but different initial vectors (IV) to encrypt different files. Then encapsulate the encrypted files into one or more package files (such as ZIP) for easy transmission. On the receiving end, the keys and algorithms required for decryption need to be pre-configured so that they can be correctly decrypted after receiving the file. When the encrypted file package arrives at the receiving end, use the corresponding decryption library and key to perform the decryption operation. The decrypted file needs to be automatically or manually moved to the specified configuration folder.

[0036] In addition, during the data distribution process, table locking caused by transaction blocking will trigger an "avalanche" effect. Therefore, when table locking or data distribution failure occurs, quickly locate the cause and location of the problem, configure the processing strategy according to the actual situation, and open up the data flow in time. According to the defined rules, the current table locking behavior can be automatically unlocked to perform data distribution operations.

[0037] Figure 2 An example diagram of an application scenario of the method of the present invention is shown.

[0038] The present invention realizes the real-time packaging of structured and unstructured data based on a single network gateway through a management method of digital detection data transmission across networks, and realizes data transmission through a data integration interface, and then parses the received packaged data according to a predetermined sequence through the receiving end network, restores the data to a specified data structure, and encrypts the transmitted data to ensure data security. This method effectively adapts to the problem of cross-network data ferrying of multiple databases, reduces the difficulty of system development and operation, quickly adapts to model changes in a configurable manner and solves the data transmission problems caused thereby, and monitors the quality of data transmission by packaging abnormal data, thereby improving the adaptability of the system.

[0039] Embodiment 2

[0040] like Figure 3 As shown, the embodiment of the present invention provides a digital detection data transmission system based on a network gate, including the following modules:

[0041] Data management module 61, used to define basic parameters, including: data interface address, data interface configuration rules, file access path and various parameters related to data flow transaction scheduling;

[0042] The data model initialization module 62 is used to define the data model and configure and manage the model attribute information that needs to be driven at both ends through the model management page, including: execution sequence, model name, model content description, model keywords, and corresponding buffer model;

[0043] The data capture module 63 is used to start the transaction flow scheduling, and capture the transmitted data flow in a transactional manner; the data generated in different time periods in various database tables are processed in an asynchronous manner, and the original time sequence of the data is retained after being captured;

[0044] The data transmission module 64 is used to encrypt and package the captured data, initiate the transmission of the data packet to be transmitted by calling the one-way network gate interface of the enterprise on the industrial control network side, and receive the data by calling the service interface through the one-way network gate on the confidential network side;

[0045] The data distribution module 65 is used to distribute data in a synchronous manner according to the original timing when the data is captured; distributed data distribution is used to meet the synchronization requirements of large-volume data.

Claims

1. A digital detection data transmission method based on a network gate, characterized in that: include: Step S1: define basic parameters, including: data interface address, data interface configuration rules, file access path, and various parameters related to data flow transaction scheduling; Step S2: define the data model, and configure and manage the model attribute information that needs to be driven at both ends through the model management page, including: execution sequence, model name, model content description, model keywords, and corresponding buffer model; Step S3: Start transaction flow scheduling, and capture the transmitted data flow in a transactional manner; use an asynchronous method to uniformly process the data generated in different time periods in various database tables, while ensuring that the original time sequence of the data can be retained after being captured; Step S4: encrypt and package the captured data, initiate transmission of the data packet to be transmitted by calling the enterprise's one-way network gate interface on the industrial control network side, and receive data by calling the service interface through the one-way network gate on the confidential network side; Step S5: Distribute data in a synchronous manner according to the original timing when the data is captured; use distributed data distribution to meet the synchronization requirements of large-volume data.

2. A digital detection data transmission system based on a network gate, characterized in that: Includes the following modules: The data management module is used to define basic parameters, including: data interface address, data interface configuration rules, file access path, and various parameters related to data flow transaction scheduling; The data model initialization module is used to define the data model. Through the model management page, the model attribute information that needs to be driven at both ends is configured and managed, including: execution sequence, model name, model content description, model keywords, and corresponding buffer model; The data capture module is used to start transaction flow scheduling and capture the transmitted data flow in a transactional manner. It uses an asynchronous method to uniformly process the data generated in various database tables at different time periods, while ensuring that the original time sequence of the data is retained after being captured. The data transmission module is used to encrypt and package the captured data, initiate the transmission of the data packets to be transmitted by calling the one-way network gate interface of the enterprise on the industrial control network side, and receive the data by calling the service interface through the one-way network gate on the confidential network side; The data distribution module is used to distribute data in a synchronous manner according to the original timing when the data is captured; distributed data distribution is used to meet the synchronization needs of large-volume data.

Images