Data transmission system and data transmission method of raw material field intelligent system

Abstract

The invention provides a data transmission system and a data transmission method of a raw material field intelligent system, and the method comprises the steps: enabling the raw material field intelligent system to be connected with a gatekeeper when the gatekeeper is not connected with an external system, generating a to-be-transmitted data file through monitoring a log file of a database, and transmitting the to-be-transmitted data file to a first designated position of the gatekeeper; the gatekeeper ferries the to-be-transmitted data file from the first specified position to a second specified position; wherein the first specified position is physically isolated from the second specified position; and the external system is connected with the gatekeeper when the gatekeeper is not connected to the raw material field intelligent system, and obtains the data file to be transmitted from the second specified position. Physical isolation between the raw material field intelligent system and the external system can be achieved through the gatekeeper, network protection of the raw material field intelligent system is enhanced, and meanwhile relevant data in the raw material field intelligent system can be safely transmitted to the external system; and powerful data support is provided for digital and intelligent construction of enterprises.

Description

Technical Field

[0001] The present invention relates to the technical field of raw material yard data transmission. Specifically, it relates to a data transmission system and a data transmission method for a raw material yard intelligent system. Background Art

[0002] The establishment of a raw material yard intelligent system is an important cornerstone for steel enterprises to achieve intelligence and digitization. Relevant data, including production, energy, and environmental protection information, are tracked, collected, analyzed, and stored in the system database through this system.

[0003] Currently, in order to achieve functional goals such as remote monitoring and data analysis, the raw material yard intelligent system usually needs to perform data interaction with external devices or networks. However, in traditional raw material yard intelligent systems, their external communication mechanisms lack clear definitions, and various communication activities mostly rely on business requirements, which can easily cause many problems such as IP address conflicts, port occupations, and network storms triggered thereby during actual operation. In addition, in order to facilitate access to and processing of the stored data, many control systems adopt the method of directly connecting to the database to achieve data extraction. This approach exposes the database directly to the external environment, thus facing security risks of being externally attacked.

[0004] In view of the above situation, how to build a highly secure data transmission path for the raw material yard intelligent system has become a technical problem to be solved urgently. Summary of the Invention

[0005] In view of this, the purpose of the present invention is to provide a data transmission system and a data transmission method for a raw material yard intelligent system, which are used to build a highly secure data transmission path for the raw material yard intelligent system and provide the security of data transmission. To achieve the above purpose, the technical solutions adopted by the present invention are as follows:

[0006] In the first aspect, the present invention provides a data transmission method for a raw material yard intelligent system. The method includes: when the network gateway is not connected to the external system, the raw material yard intelligent system connects to the network gateway, generates a data file to be transmitted by listening to the log file of the database, and transmits it to the first designated position of the network gateway; the network gateway ferries the data file to be transmitted from the first designated position to the second designated position; wherein, the first designated position and the second designated position are physically isolated; when the network gateway is not connected to the raw material yard intelligent system, the external system connects to the network gateway and obtains the data file to be transmitted from the second designated position.

[0007] In the second aspect, the present invention provides a data transmission system, which is used to execute the data transmission method for the raw material yard intelligent system as described in the first aspect.

[0008] The data transmission system provided by the embodiments of the present invention and the data transmission method of the intelligent raw material yard system, compared with the existing data transmission methods, this method realizes the physical isolation between the intelligent raw material yard system and the external system through a network gateway. During data transmission, the network gateway is always connected to one of the systems, that is, when the intelligent raw material yard system issues a data file to be transmitted, the network gateway is only connected to the intelligent raw material yard system, and when the external system acquires data, the network gateway is only connected to the external system. In this way, the intelligent raw material yard system and the external system are completely physically isolated through the network gateway. The network gateway ferries the data file to be transmitted from the first specified location to the second specified location in a protocol-free manner, strengthening the network protection of the intelligent raw material yard system. At the same time, it can also enable the relevant data in the intelligent raw material system to be safely transmitted to the external system, providing strong data support for the digital and intelligent construction of the enterprise.

[0009] To make the above objects, features, and advantages of the present invention more obvious and understandable, the following specifically enumerates preferred embodiments and, in conjunction with the accompanying drawings, makes a detailed description as follows. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] To more clearly illustrate the technical solutions of the embodiments of the present invention, the following will briefly introduce the drawings required to be used in the embodiments. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as limiting the scope. For those of ordinary skill in the art, without creative efforts, other relevant drawings can also be obtained based on these drawings.

[0011] Figure 1 is the architecture of the existing data transmission system for the intelligent raw material yard system;

[0012] Figure 2 is the architecture diagram of a data transmission system provided by the embodiments of the present invention;

[0013] Figure 3 is the schematic flowchart of the data transmission method of the intelligent raw material yard system provided by the embodiments of the present invention;

[0014] Figure 4 is an example diagram of the overall data transmission process of the intelligent raw material yard system provided by the embodiments of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0015] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Usually, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0016] Accordingly, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative efforts fall within the scope of protection of the present invention.

[0017] It should be noted that relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements, but also includes other elements not expressly listed, or elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "including a..." does not exclude the existence of additional identical elements in the process, method, article or device including the said element.

[0018] Please refer to Figure 1 , Figure 1 which is an existing data transmission system architecture for the raw material yard intelligent system, including the raw material yard basic automation system, the raw material yard intelligent system, the raw material yard firewall and the external system, and the arrows indicate the data flow direction.

[0019] In Figure 1 the shown raw material yard basic automation system, the PLC (Programmable Logic Controller) serves as the core control unit and can flexibly collect or distribute data according to business requirements. This system can interact with the raw material yard intelligent control system through two communication methods, namely OPC and Socket. The raw material yard intelligent system not only collects data from the raw material yard basic automation system, but also processes these data and conducts calculation and analysis, and all data are stored in the database of the raw material yard intelligent control system, such as relational databases like Oracle or MySQL. In order to achieve network isolation between the raw material yard intelligent system and the external system, a raw material yard firewall is also set up, and by configuring the internal and external address mapping, the external system (such as the company's ERP system) can obtain the required data from the database of the raw material yard intelligent system.

[0020] The inventor has found through research that: Figure 1 the following defects exist in the external data transmission method of the shown raw material yard intelligent system:

[0021] Defect 1: The original intelligent system of the raw material yard lacks a clear communication boundary with external systems in its design. Currently, this system allows communication with any IP address and, to adapt to different business requirements, opens multiple ports for data reception and transmission. This design has led to problems such as IP conflicts, port occupation, and network storms, increasing the instability of the system.

[0022] Defect 2: The intelligent system of the raw material yard lacks physical isolation from external systems, resulting in the database being directly exposed to the external network. Even the Web interface of the intelligent control system can be directly accessed via the Internet, which poses significant data security risks to the system.

[0023] Defect 3: When transmitting text data through the Socket communication protocol, it is unable to effectively transfer complex information such as the metadata of the database (such as table structure, field definitions, etc.), data table changes, and transaction changes.

[0024] Considering the above security issues existing in the existing data transmission methods of the intelligent system of the raw material yard. Based on this, the embodiments of the present invention provide an improved system architecture. Please refer to Figure 2 , Figure 2 FIG. shows an architecture diagram of a data transmission system provided by an embodiment of the present invention, including an intelligent system 210 of the raw material yard, a network gateway 220, and an external system 230. The arrows indicate the data flow direction. The network gateway 210 is only connected to one of the intelligent system 210 of the raw material yard and the network gateway 220 each time.

[0025] It should be understood that Figure 2 FIG. shows that both the intelligent system 210 of the raw material yard and the external system 230 are connected to the network gateway 220. This display method is intended to simplify and clearly present the system structure and does not constitute a limitation on the actual connection relationship. In actual deployment, the connection method may be adjusted according to specific requirements and security policies, and is not limited to the configuration shown in the figure.

[0026] It can be understood that Figure 2 the structure shown in FIG. is only schematic. The data transmission system 20 may further include more or fewer components than those shown in Figure 2 FIG., or have a different configuration from that shown in Figure 2 FIG. Figure 2 The systems shown in FIG. may be implemented using hardware, software, or a combination thereof.

[0027] The intelligent system 210 of the raw material yard is used to connect to the network gateway 220 when the network gateway 220 is not connected to the external system 230, generate a data file to be transmitted by listening to the log file of the database, and transmit it to the first specified location of the network gateway.

[0028] The air gap 220 is used to ferry the data file to be transmitted from the first specified location to the second specified location; wherein, the first specified location and the second specified location are physically isolated;

[0029] The external system 230 is used to connect to the air gap 220 when the air gap 220 is not connected to the raw material yard intelligent system 210, and obtain the data file to be transmitted from the second specified location.

[0030] In the embodiment of the present invention, the first specified location is the communication side between the raw material yard intelligent system 210 and the air gap 220, which can be understood as the "inner side of the air gap". The second specified location can be understood as the outer side of the air gap, that is, the communication side with the external system 230.

[0031] In the embodiment of the present invention, the air gap 220 is responsible for physically isolating the raw material yard intelligent system 210 and the external system 230, thereby avoiding the risk of database intrusion or information leakage. At any moment, the air gap 310 is only connected to one of the raw material yard intelligent system 210 and the external system 230. Through the air gap 220, the data file can be ferried from the first specified location to the second specified location in a protocol-free manner for the external system 230 to obtain the data.

[0032] The data transmission system 20 provided by the embodiment of the present invention realizes physical isolation between the raw material yard intelligent system 210 and the external system 230 through the air gap 220. During the data transmission process, the air gap 220 is always only connected to one of the systems: when the raw material yard intelligent system 210 issues the data file to be transmitted, the air gap 220 is only connected to this system; and when the external system 230 needs to obtain data, the air gap 220 switches to only connect to the external system 230. This mechanism ensures that there is always strict physical isolation between the raw material yard intelligent system 210 and the external system 230. The air gap 220 ferries the data file to be transmitted from the first specified location (i.e., the inner side of the air gap) to the second specified location (i.e., the outer side of the air gap) in a protocol-free manner for the external system 230 to obtain the data. Through the data transmission system 20 provided by the embodiment of the present invention, not only the network protection of the raw material yard intelligent system 220 is strengthened, but also the relevant data can be safely and reliably transmitted to the external system 230, providing strong data support for the digital and intelligent construction of the enterprise, and at the same time significantly improving the security and controllability of data transmission.

[0033] Next, the embodiment of the present invention will introduce in detail Figure 2 each component in the shown system architecture.

[0034] In the raw material yard intelligent system 210, in the embodiments of the present invention, considering the original database tracking mode, the regular polling and service-triggered query methods are adopted, which have high latency and low efficiency. There is no effective solution for processing operations such as deletion and update, and it is not suitable for transmitting data from the transactional database to the big data processing platform.

[0035] To solve the above problem, the raw material yard intelligent system 210 in the embodiments of the present invention can use the FlinkCDC (Change Data Capture) technology to monitor the database operation logs of the intelligent control system. When the database generates change data, corresponding data files will be generated. These changes can include, but are not limited to, database metadata changes, data table changes, transaction changes, etc., so as to generate data files to be transmitted.

[0036] In an embodiment of the present invention, in view of the fact that the regular truncation characteristic of the operation logs in the database may cause the monitoring to be interrupted, thus missing some changes in the operation logs, the raw material yard intelligent system 210 in the embodiments of the present invention realizes an efficient monitoring mechanism through steps a1 to a3. The specific description is as follows:

[0037] Step a1: Monitor the log file through an asynchronous call program;

[0038] Step a2: When a change occurs in the monitored log file, generate a change data file as the data file to be transmitted;

[0039] In the embodiments of the present invention, the raw material yard intelligent system 210 adopts an asynchronous call program, which can effectively prevent the interruption phenomenon during the data tracking process and can detect the missing change data in a timely manner. To achieve this goal, the process of the raw material yard intelligent system 210 monitoring the log file through the asynchronous call program can be:

[0040] The first step: Monitor the log file through the first monitoring process;

[0041] The second step: When the first monitoring process is interrupted, switch to the second monitoring process to rescan the newly generated log file and monitor it.

[0042] It can be understood that during the log truncation operation, if the first monitoring process is interrupted, the raw material yard intelligent system 210 will automatically start the second monitoring process. This process will rescan the newly generated log file and continue to monitor in real time to ensure the continuity and integrity of data tracking and change detection, thus maintaining the efficient operation of the system.

[0043] In an embodiment of the present invention, after the second monitoring process scans a newly generated log file, the second monitoring can detect whether there is any missing change data. If so, the raw material yard intelligent system 210 can generate a change data file for the missing change.

[0044] In step a2, once the log file in the database of the raw material yard intelligent system 210 changes, an event will be triggered. This event usually includes the type of change (such as file addition, modification, deletion) and the timestamp of the change. The raw material yard intelligent system 210 generates a change data file based on this information.

[0045] In an embodiment of the present invention, the raw material yard intelligent system 210 can use the FlinkCDC technology to capture the changes in the log file in real time and generate a change data file in the order of occurrence. In this way, the system can accurately record and process every data change.

[0046] In the embodiment of the present invention, the format of the change data file can be selected according to specific requirements, including but not limited to the following formats: JSON, XML, TXT, etc., so that the system can adapt to different application scenarios and technical requirements.

[0047] In an embodiment of the present invention, considering that there may be multiple changes of the same type at the same time, but the business requirements of each change are different. For example, 10 "insert" operations occur in the log file at the same time, but the business requirements of each "insert" operation are not the same. If the traditional file generation method is used, it may cause the file names of the change files for these 10 "insert" operations to be the same and result in file overwriting.

[0048] To solve the above problem, in the embodiment of the present invention, the raw material yard intelligent system 210 adopts a preset identifier generation algorithm to generate a unique file name for each change data file. Specifically, no matter how many change operations of the same type occur at the same time (such as multiple "insert" operations), the system can ensure that each change data file has a unique file name. In this way, even if the operation types are the same but the business requirements are different, non-repeating file names can be generated, effectively avoiding the risk of file overwriting and ensuring that the specific business requirements of each operation can be accurately recorded and distinguished.

[0049] In one embodiment of the present invention, the preset identification generation algorithm may be the Snowflake Algorithm. It is a solution for generating distributed primary key IDs. After generation, it is a 64-bit long numerical value. The components introduce timestamps and basically maintain increment. Through this algorithm, unique ID naming can be generated, and the intelligent raw material yard system 210 can obtain non-repeating file names.

[0050] Through the above implementation, after the intelligent raw material yard system 210 generates the data to be transmitted, it can connect to the network gateway 220 when the network gateway is not connected to the external system 230, and then securely transmit the file to be transmitted to the first designated location of the network gateway 220.

[0051] In one embodiment of the present invention, the intelligent raw material yard system 210 can send the data file to be transmitted to the first designated location of the network gateway through the File Transfer Protocol (FTP). For example, as an example, the File Transfer Protocol can be SFTP (SSH File Transfer Protocol, Secure File Transfer Protocol), which is a secure version of the File Transfer Protocol and is also part of the SSH protocol. It can easily perform data transmission and data access through the secure SHELL (SSH) data stream. Thus, the secure data transmission between the intelligent raw material yard system 210 and the network gateway 220 is completed.

[0052] Next, the network gateway 220 in the embodiments of the present invention will be introduced.

[0053] In the embodiments of the present invention, after the network gateway 220 obtains the data file to be transmitted, it can ferry the data file to be transmitted from the first designated location to the second designated location.

[0054] In the embodiments of the present invention, the first designated location and the second designated location in the network gateway 220 are physically isolated. Inside the network gateway 220, the data flow direction can be: from the intelligent raw material yard system 210 to the first designated location, and from the second designated location to the external system 230.

[0055] The first designated location and the second designated location can be determined based on the actual system architecture of the network gateway 220. For example, as an example, the network gateway 220 may include two sets of systems, and these two sets of systems are physically isolated. Then the first designated location is the system communicating with the intelligent raw material yard system, and the second designated location can be the system communicating with the external system 230.

[0056] This physically isolated design ensures the security of data during transmission, prevents unauthorized access to the intelligent raw material yard system 2210 and potential security threats, and guarantees the security of the entire system.

[0057] In the embodiment of the present invention, after the air gap gateway 220 obtains the data file to be transmitted, it can ferry the file to the second specified location, that is, outside the air gap gateway, in a protocol-free manner, so that the external system 230 can parse the data file, and the entire data security transfer process is completed.

[0058] Next, the external system 230 in the embodiment of the present invention will be introduced.

[0059] In the embodiment of the present invention, the external system 230 is connected to the air gap gateway 220 when the air gap gateway 220 is disconnected from the intelligent system 210 of the raw material yard, and can obtain the data file to be transmitted from the second specified location.

[0060] It can be understood that the external system 230 can parse and consume the data file to be transmitted, and transmit the processed data to the private cloud application, and then use tools such as Kafka and PostgreSQL to complete data collection and storage.

[0061] In an embodiment of the present invention, technicians can also obtain data from the enterprise private cloud application through technical means such as API (Application Programming Interface) and JDBC (Java Database Connectivity), and develop various types of application programs using these data.

[0062] In the embodiment of the present invention, the external system 230 can communicate with the air gap gateway 220 according to business requirements, which can avoid data omission problems caused by limitations such as collection frequency and single collection data volume.

[0063] The data transmission system provided by the embodiment of the present invention has the following advantages:

[0064] First of all, the intelligent system 210 of the raw material yard can use the FlinkCDC technology to monitor and capture the changes (additions, deletions, modifications, etc.) of the database, capture these changes in the order of occurrence, and generate various formats of files as needed. During the monitoring process, the intelligent system 210 of the raw material yard adopts a dual-process mechanism and timely monitors the changes through asynchronous calls to avoid missing some changes due to the truncation operation of the log file. The intelligent system 210 of the raw material yard securely transmits the data file to the inside of the air gap gateway using the text transfer protocol, and the air gap gateway 220 ferries the data file to the outside of the air gap gateway in a protocol-free manner for the external system 230 to parse and consume.

[0065] During the entire transmission process, the network gate 220 is always connected to one of the systems. That is, when the intelligent system 210 of the raw material field sends data files, the network gate 220 is only connected to the intelligent system 210 of the raw material field, and when the external system 230 obtains data, the network gate 220 is only connected to the external system 230. In this way, the intelligent system 210 of the raw material field and the external system 230 are completely physically isolated through the network gate. There is no information exchange based on the protocol inside and outside the network gate, but only non-protocol ferrying in the form of data files. This measure strengthens the network protection of the intelligent system of the raw material field, and allows the production, energy, environmental protection, cost and other related data related to tracking, collection and analysis in the intelligent system 210 of the raw material to be safely transmitted to the private cloud application of the external system 230, providing strong data support for the digitalization and intelligent construction of enterprises.

[0066] Based on Figure 2 For the proposed system architecture, see Figure 3 , Figure 3 The schematic flow chart of the data transmission method of the intelligent system of the raw material field provided by the embodiment of the present invention includes steps S301 to S303, which are described as follows:

[0067] S301: the intelligent system of the raw material yard connects to the network gate when the network gate is not connected to the external system, generates a data file to be transmitted by monitoring the log file of the database, and transmits it to the first designated location of the network gate;

[0068] S302: The network gate transfers the data file to be transmitted from the first designated location to the second designated location; wherein the first designated location is physically isolated from the second designated location;

[0069] S303: The external system connects to the network gate when the network gate is not connected to the intelligent system of the raw material yard, and obtains the data file to be transmitted from the second designated location.

[0070] Through the scheme shown in the above steps S301 to S303, the intelligent system of the raw material field in the embodiment of the present invention can generate a data file to be transmitted by monitoring the log file and send it to the first designated location of the network gate. At this time, the network gate is only connected to the intelligent system of the raw material field. After receiving the data file to be transmitted, the network gate can ferry the data file to the first designated location in a non-protocol manner so that the external system can obtain the data from the second designated location. At this time, the network gate is only connected to the external system, and the entire data transmission process strengthens the network protection of the intelligent system of the raw material field, allowing the production, energy, environmental protection, cost and other related data related to tracking, collection and analysis in the intelligent raw material system to be safely transmitted to the private cloud application, providing strong data support for the digitalization and intelligent construction of the enterprise.

[0071] In an alternative embodiment, a data file to be transmitted is generated by listening to the log file of the database, including: listening to the log file through an asynchronous calling program; when a change occurs in the log file, generating a change data file as the data file to be transmitted.

[0072] In an alternative embodiment, the data transmission method provided by the embodiments of the present invention may further include: generating a unique file name for the data file to be transmitted by using a preset identification generation algorithm.

[0073] In an alternative embodiment, listening to the log file through an asynchronous calling program includes: listening to the log file through a first listening process; when the first listening process is interrupted, switching to a second listening process to re-scan and listen to the newly generated log file.

[0074] In an alternative embodiment, the data transmission method provided by the embodiments of the present invention may further include: after the second listening process scans the newly generated log file, detecting whether there is any missing change data, and if so, generating a change data file corresponding to the missing change data.

[0075] In an alternative embodiment, the data transmission method provided by the embodiments of the present invention may further include: generating a change data file by listening to the log file through a change data capture technology.

[0076] In an alternative embodiment, the data transmission method provided by the embodiments of the present invention may further include: the raw material yard intelligent system transfers the data file to be transmitted to a first specified location through a file transfer protocol.

[0077] In an alternative embodiment, the data transmission method provided by the embodiments of the present invention may further include: an external system parses and consumes the data file to be transmitted, and transmits the processed data to a private cloud application.

[0078] In an alternative embodiment, the preset identification generation algorithm provided by the embodiments of the present invention is the Snowflake algorithm.

[0079] For a better understanding of the data transmission method provided by the embodiments of the present invention, please refer to Figure 4 , Figure 4 which shows an example of the overall data transmission process of the raw material yard intelligent system in the embodiments of the present invention.

[0080] From Figure 4As can be seen, the data transmission method adopted in the embodiments of the present invention utilizes the FlinkCDC technology to accurately monitor and capture various changes in the database (including operations such as addition, deletion, and modification), and obtain the changed data. The system captures these changes in the order of their occurrence and generates changed data files in various formats, such as JSON, XML, TXT, etc. Subsequently, the Snowflake algorithm is used to generate unique file names for these changed data files, and then the SFTP protocol is used to securely transmit the data files to the first designated location of the network gateway. The network gateway ferries the data files accurately from the first designated location to the second designated location in a protocol-free manner. The external system can parse these data files at the second designated location and access the parsed data into the private cloud application, and various applications can be generated based on these data.

[0081] Generally speaking, the above data transmission method provided by the embodiments of the present invention has the characteristics of high efficiency and security, and can be widely applied to various scenarios with strict requirements for data secure transmission, including enterprises, departments, and scientific research institutions, etc.

[0082] In several embodiments provided in the present application, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architectures, functions, and operations of devices, methods, and computer program products according to multiple embodiments of the present invention. In this regard, each block in the flowchart or block diagram may represent a module, a program segment, or a part of code, and the module, program segment, or part of code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order from that marked in the accompanying drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and / or flowchart, and the combination of blocks in the block diagram and / or flowchart, can be implemented by a dedicated hardware-based system for performing the specified functions or actions, or can be implemented by a combination of dedicated hardware and computer instructions.

[0083] In addition, each functional module in various embodiments of the present invention can be integrated together to form an independent part, or each module can exist separately, or two or more modules can be integrated to form an independent part.

[0084] When the above-mentioned functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or a part of this technical solution can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), magnetic disks, or optical discs that can store program codes.

[0085] The foregoing are only the preferred embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention may have various changes and modifications. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims

1. A data transmission method for a raw material field intelligent system, characterized in that: The method comprises: The intelligent system of the raw material yard connects to the network gate when the network gate is not connected to the external system, generates a data file to be transmitted by monitoring the log file of the database, and transmits it to the first designated location of the network gate; The network gate transfers the data file to be transmitted from the first designated location to the second designated location; wherein the first designated location is physically isolated from the second designated location; The external system connects to the gateway when the gateway is not connected to the intelligent system of the raw material yard, and obtains the data file to be transmitted from the second designated location.

2. The data transmission method of the intelligent system of the raw material field according to claim 1 is characterized in that: Generate data files to be transmitted by monitoring the log files of the database, including: Monitor the log file by asynchronously calling a program; When a change is detected in the log file, a change data file is generated as the data file to be transmitted.

3. The data transmission method of the intelligent system of the raw material field according to claim 2 is characterized in that: The method further comprises: A unique file name is generated for the data file to be transmitted using a preset identifier generation algorithm.

4. The data transmission method of the intelligent system of the raw material field according to claim 2 is characterized in that: Monitoring the log file through an asynchronous calling program includes: Monitoring the log file through a first monitoring process; When the first monitoring process is interrupted, the second monitoring process is switched to rescan the newly generated log file and perform monitoring.

5. The data transmission method of the intelligent system of the raw material field according to claim 4 is characterized in that: The method further comprises: After the newly generated log file is scanned by the second monitoring process, it is detected whether there is any missing change data. If so, the change data file corresponding to the missing change data is generated.

6. The data transmission method of the intelligent system of the raw material field according to claim 2 is characterized in that: The method further comprises: The log file is monitored by a change data capture technology to generate a change data file.

7. The data transmission method of the intelligent system of the raw material field according to claim 1 is characterized in that: The method further comprises: The raw material yard intelligent system transfers the data file to be transferred to the first designated location through a file transfer protocol.

8. The data transmission method of the intelligent system of the raw material field according to claim 1 is characterized in that: The method further comprises: The external system parses and consumes the data file to be transmitted, and transmits the processed data to the private cloud application.

9. The data transmission method of the intelligent system of the raw material field according to claim 3 is characterized in that: The preset identifier generation algorithm is the snowflake algorithm.

10. A data transmission system, characterized in that: The data transmission system is used to execute the data transmission method of the intelligent raw material field system as described in any one of claims 1 to 9.

Images