Oil and gas pipeline multi-service data dynamic transmission and loading method and system

Abstract

The application relates to a dynamic transmission and loading method and system of oil and gas pipeline multi-service data; the dynamic transmission and loading method comprises the following steps: automatically configuring the number of data transmission channels, the number of concurrents and the number of threads according to the size and data generation speed of the oil and gas pipeline multi-service data, collecting or synchronizing the oil and gas pipeline multi-service data to obtain dynamic synchronization data; storing the dynamic synchronization data into a dynamic data storage module; reading the dynamic synchronization data and providing a dynamic data sharing interface; and loading the dynamic synchronization data by accessing the dynamic data sharing interface by an upper-layer application, so that the data processing work of the oil and gas pipeline multi-service data is realized. Through the setting of the dynamic data loading intermediate service, the problems of unified collection of high timeliness, large data volume and multi-structure multi-source heterogeneous service system data can be solved, the upper-layer application can be designed simply, the dynamic loading pressure of the upper-layer application is relieved, and the post-maintenance of the upper-layer application is facilitated.

Description

Technical Field

[0001] This invention relates to the field of digital pipeline construction technology, specifically to a method and system for dynamic transmission and loading of multi-service data for oil and gas pipelines. Background Technology

[0002] With the increasing popularity and application of computer technology, driven by the wave of informatization and digitalization, oil and gas pipeline companies have built more and more multi-business systems, resulting in data silos between various business systems of oil and gas pipelines, slow loading of dynamic data, and many problems often cannot be dealt with in a timely manner during operation.

[0003] Traditional dynamic data loading technologies suffer from a simplistic architecture, limited single-machine resources, limited scalability, small capacity, and low performance. With the maturity of IoT technology and the development of information technology, the information systems and IoT systems built in the oil and gas pipeline industry have effectively supported pipeline operation and maintenance management and are widely used in the industry, generating a large amount of real-time dynamic data such as detection and anomaly alarms. However, the construction of these systems did not consider the need for a holistic and comprehensive observation of pipeline operation and maintenance from a global perspective as business needs evolve. This has inadvertently created many siloed systems, resulting in a large amount of isolated dynamic data. For example, in a pipeline company's SCADA system, there are typically 6000+ data points, generating over 6000 data entries per second, accumulating to approximately 1.7GB of data per hour.

[0004] To achieve the goals of smart pipeline construction, the first step is to achieve comprehensive data unification. However, relying on traditional, single-structure dynamic data loading technology and supporting traditional relational databases is insufficient to handle the loading of dynamic data characterized by high timeliness, large volume, and diverse structures.

[0005] Patent document CN103077134A, entitled "A Method and Apparatus for Implementing Multi-Pipe Data Transmission in an Embedded System," describes a method and apparatus for implementing multi-pipe data transmission in an embedded system. This method achieves multi-pipe data transmission by creating dynamic channels between entities in the host network and setting a corresponding transmission buffer for each pipe, thereby avoiding potential crossovers between pipes, reducing data transmission errors between entities, and improving data transmission reliability. Since dynamic pipes can be dynamically created and deleted, long-term occupation of management resources is avoided, saving host resources. Furthermore, this apparatus enables centralized control, supports network topologies, is simple to implement, easy to deploy, supports multiple underlying data carriers, is suitable for various application scenarios, has strong error handling capabilities, and is compatible with multiple hardware platforms. Patent document CN110147356A, entitled "Data Transmission Method and Apparatus," discloses a data transmission method and apparatus. The method includes: custom-collecting different data sources to obtain different collected data, and parsing the different collected data into data packets of a unified data format; writing the unified data packets into a data pipeline for caching; reading the cached data packets in the data pipeline and writing the read data packets into a downstream data storage service to achieve data transmission; and limiting the data transmission rate during the data transmission process based on a preset peak traffic value of the downstream data storage service.

[0006] While these technologies can all achieve dynamic data transfer and loading, thus improving loading speed, they all involve direct interaction between upper-layer applications and multi-service database systems, leading to application data redundancy and bloated design. They cannot truly achieve efficient loading and timely synchronization of dynamic data. Summary of the Invention

[0007] To address the technical problems of data silos and slow dynamic data loading between various business systems in oil and gas pipelines, this invention provides a method and system for dynamic transmission and loading of multi-business data in oil and gas pipelines.

[0008] To address the aforementioned technical problems, this invention provides a method for dynamic transmission and loading of multi-service data in oil and gas pipelines, the technical solution of which is as follows:

[0009] A method for dynamic transmission and loading of multi-service data in oil and gas pipelines includes the following steps:

[0010] Based on the size and data generation speed of the multi-service data of the oil and gas pipeline, the number of data transmission channels, the number of concurrency, and the number of threads are automatically configured;

[0011] Based on the automatically configured number of data transmission channels, the number of concurrent connections, and the number of threads, the multi-service data of the oil and gas pipeline is collected or synchronized to obtain dynamic synchronization data;

[0012] The dynamically synchronized data is stored in the dynamic data storage module;

[0013] The system reads the dynamic synchronization data from the dynamic data storage module and provides a dynamic data sharing interface to external systems.

[0014] Upper-layer applications load the dynamically synchronized data by accessing the dynamic data sharing interface, thereby enabling data processing of multi-service data for the oil and gas pipeline; specifically, the data processing involves data transmission or data loading.

[0015] The beneficial effects of this invention are as follows: This invention acquires multi-service data of oil and gas pipelines using data synchronization technology, adjusts the data structure of this multi-service data to an acyclic graph data topology, and then uses an intermediate storage module and a dynamic data storage module to synchronize the dynamically synchronized data. Upper-layer applications can then access this multi-service data through a unified data sharing interface. Through a multi-threaded, multi-channel design, machine resources are fully utilized, greatly improving data synchronization speed and providing a reliable technical foundation for the implementation of more application scenarios. Furthermore, by dynamically storing data and sharing the stored dynamic data, a consistent interface protocol is exposed, helping upper-layer applications reduce code, simplify design, and facilitate future maintenance.

[0016] Based on the above technical solution, the present invention can be further improved as follows.

[0017] Furthermore, based on the automatically configured number of data transmission channels, the number of concurrent connections, and the number of threads, the multi-service data of the oil and gas pipeline is collected or synchronized to obtain dynamically synchronized data, including the following steps:

[0018] The oil and gas pipeline multi-service data is collected or synchronized according to the automatically configured number of data transmission channels, the number of concurrent connections, and the number of threads.

[0019] A directed acyclic graph (DAG) data topology is established, and the association structure of the multi-business data of the oil and gas pipeline is established using the DAG data topology to obtain the dynamic synchronization data.

[0020] The beneficial effect of adopting the above-mentioned further solution is that by using the directed acyclic graph data topology structure to construct multi-step business associations, dynamic data can be processed during the flow process to fulfill the need for additional data associations.

[0021] Furthermore, during the collection or synchronization of multi-service data from the oil and gas pipelines, corresponding data processing logs are also generated. The advantage of this further approach is that generating data processing logs records the data collection and synchronization processes, enabling traceable and systematic tracking.

[0022] Furthermore, storing the dynamically synchronized data into the dynamic data storage module includes the following steps:

[0023] Using a dynamic partitioning method, the dynamic synchronization data is partitioned according to the time field, and the partitioned dynamic synchronization data is stored in the dynamic data storage module.

[0024] The advantage of adopting the above-mentioned further solution is that by using dynamic partitioning technology, the dynamically synchronized data is partitioned according to the time field, which enables the upper-layer application to load the dynamically synchronized data efficiently.

[0025] Furthermore, storing the partitioned dynamic synchronization data in the dynamic data storage module includes the following steps:

[0026] The dynamically synchronized data after partitioning is stored in the dynamic data storage module in a columnar storage format.

[0027] The advantages of adopting the above-mentioned further solutions are that columnar storage format can read only the relevant columns during queries, and the columns can be directly used as indexes, which is very efficient, while row-based storage must read the entire row. Projection operations in columnar storage are also very efficient. In cases of sparse data, the compression ratio is much higher than that of row-based storage, and it is even possible to consider pre-joining related tables to completely avoid projection operations.

[0028] Furthermore, storing the dynamically synchronized data in the dynamic data storage module also includes the following steps: storing the dynamically synchronized data and the data processing log in the dynamic data storage module.

[0029] Furthermore, based on the size and data generation speed of the multi-service data from the oil and gas pipeline, the number of data transmission channels, concurrency, and threads are automatically configured, including the following steps:

[0030] Based on the size and data generation speed of the multi-service data of the oil and gas pipeline, the number of data transmission channels, concurrency, and threads of each node in the distributed system are automatically configured according to the distributed system design model.

[0031] The beneficial effect of adopting the above-mentioned further solution is that it utilizes the distributed design system concept, multi-node response, and data replication architecture. This enables the technology to easily handle the transmission or loading of massive amounts of dynamic data, greatly satisfying the timeliness requirements of upper-layer applications for dynamic data loading.

[0032] Furthermore, reading the dynamic synchronization data from the dynamic data storage module and providing a dynamic data sharing interface to the outside includes the following steps: reading the dynamic synchronization data from the dynamic data storage module and providing the dynamic data sharing interface to the outside based on the principle of interface unification.

[0033] The beneficial effect of adopting the above-mentioned further solutions is that, based on the principle of interface consistency, less thought and effort is required to write new code that remains consistent with other parts of the system functions. The principle of interface consistency allows for the reuse of code running in other parts of the system, thereby saving development time and subsequent maintenance time, and reducing operating costs.

[0034] To address the aforementioned technical problems, this invention provides a dynamic transmission and loading system for multi-service data in oil and gas pipelines, the technical solution of which is as follows:

[0035] A dynamic transmission and loading system for multi-service data in oil and gas pipelines, comprising,

[0036] The dynamic data synchronization module is used to automatically configure the number of data transmission channels, the number of concurrent connections, and the number of threads based on the size and data generation speed of the multi-service data of the oil and gas pipeline; and to collect or synchronize the multi-service data of the oil and gas pipeline based on the automatically configured number of data transmission channels, the number of concurrent connections, and the number of threads to obtain dynamic synchronization data.

[0037] A dynamic data storage module is used to store the dynamically synchronized data;

[0038] The dynamic data sharing module is used to read the dynamically synchronized data from the dynamic data storage module and provide a dynamic data sharing interface to the outside world; wherein, the upper-layer application loads the dynamically synchronized data by accessing the dynamic data sharing interface to realize the data processing of multi-business data of the oil and gas pipeline; wherein, the data processing specifically refers to data transmission or data loading. Attached Figure Description

[0039] Figure 1 This is a flowchart illustrating a method for dynamic transmission and loading of multi-service data in oil and gas pipelines according to an embodiment of the present invention.

[0040] Figure 2 The diagram below illustrates the structural principle of a dynamic transmission and loading system for multi-service data in oil and gas pipelines, as described in this embodiment of the invention. Detailed Implementation

[0041] The principles and features of the present invention are described below with reference to the accompanying drawings. The examples given are only for explaining the present invention and are not intended to limit the scope of the present invention.

[0042] like Figure 1 As shown, this embodiment provides a method for dynamic transmission and loading of multi-service data for oil and gas pipelines, including the following steps:

[0043] S1. Based on the size and data generation speed of the multi-business data of the oil and gas pipeline, automatically configure the number of data transmission channels, the number of concurrency and the number of threads; the specific steps are as follows: based on the size and data generation speed of the multi-business data of the oil and gas pipeline, automatically configure the number of data transmission channels, the number of concurrency and the number of threads of each node of the distributed system according to the distributed system design model.

[0044] S2. Collect or synchronize the above-mentioned multi-service data of oil and gas pipelines according to the automatically configured number of data transmission channels, the number of concurrent connections, and the number of threads to obtain dynamic synchronization data.

[0045] The specific steps are as follows: collecting or synchronizing the multi-service data of the oil and gas pipeline according to the automatically configured number of data transmission channels, the number of concurrent connections, and the number of threads; establishing a directed acyclic graph data topology structure; using the directed acyclic graph data topology structure to establish a data association structure for the multi-service data of the oil and gas pipeline to obtain the dynamic synchronized data; and generating corresponding data processing logs during the collection or synchronization of the multi-service data of the oil and gas pipeline.

[0046] S3. Store the aforementioned dynamic synchronization data in the dynamic data storage module; the specific steps are as follows: using dynamic partitioning technology, partition the aforementioned dynamic synchronization data according to the time field, store the partitioned dynamic synchronization data in the aforementioned dynamic data storage module according to the columnar storage format, and store the aforementioned data processing log in the dynamic data storage module.

[0047] S4. Read the dynamic synchronization data from the dynamic data storage module and provide a consistent dynamic data sharing interface to the outside. Specifically, read the dynamic synchronization data from the dynamic data storage module and provide the dynamic data sharing interface to the outside based on the principle of interface unification.

[0048] S5. The upper-layer application loads the dynamic synchronization data by accessing the dynamic data sharing interface, thereby realizing the data processing of the multi-business data of the oil and gas pipeline; whereby data processing specifically refers to data transmission or data loading.

[0049] This invention utilizes data synchronization technology to acquire multi-service data from oil and gas pipelines. The data structure of this multi-service data is adjusted to an acyclic graph topology. The dynamically synchronized data is then synchronized using an intermediate storage module and a dynamic data storage module. Upper-layer applications can access this data through a unified data sharing interface. This invention integrates distributed, multi-threaded, and multi-channel design concepts, fully utilizing machine resources. Synchronizing 500MB of liquid level data takes only about 10 seconds, significantly improving data synchronization speed and providing a reliable technical foundation for more application scenarios. Employing distributed system design principles, the invention leverages the characteristics of distributed systems such as high availability, horizontal scalability, and horizontal deployment, enabling the dynamic data storage module to store massive amounts of historical data. When storage space is insufficient, machine resources can be added appropriately based on an estimated increase in future data volume. This invention provides a consistent interface protocol through dynamic data storage and sharing of the stored dynamic data, helping upper-layer applications reduce code, simplify design, and facilitate future maintenance.

[0050] Example 2

[0051] This embodiment provides a dynamic transmission and loading system for multi-service data in oil and gas pipelines, including:

[0052] The dynamic data synchronization module is used to automatically configure the number of data transmission channels, the number of concurrent connections, and the number of threads based on the size and data generation speed of the multi-business data of the oil and gas pipeline. It collects or synchronizes the multi-business data of the oil and gas pipeline according to the automatically configured number of data transmission channels, the number of concurrent connections, and the number of threads to obtain dynamic synchronized data. The multi-business data of the oil and gas pipeline is obtained by the data acquisition and monitoring control system from various parameters and business data of the oil and gas pipeline.

[0053] The dynamic data storage module is used to store the aforementioned dynamically synchronized data;

[0054] The dynamic data sharing module is used to read the dynamic synchronization data from the dynamic data storage module and provide a dynamic data sharing interface to the outside world. The upper-layer application loads the dynamic synchronization data by accessing the dynamic data sharing interface to realize the data processing of the multi-business data of the oil and gas pipeline. The data processing specifically refers to data transmission or data loading.

[0055] The specific setup steps and methods for the dynamic data synchronization module, dynamic data storage module, and dynamic data sharing module are as follows:

[0056] Based on the design concept of distributed systems, a dynamic data synchronization module is set up to replace the application in the previous version to directly connect with multiple business databases to obtain dynamic data sources. Based on the size of the dynamic data and the data generation speed, the number of channels, concurrency and threads are automatically configured to realize the dynamic data synchronization function.

[0057] Set up a dynamic data storage module, and by establishing a corresponding data model in the dynamic data storage module, store the synchronized dynamic data in the dynamic data storage module.

[0058] A dynamic data sharing module is set up to read dynamic data stored in the dynamic data storage module and provide a consistent dynamic data sharing interface to the outside world. Upper-layer applications can load dynamic data in real time by accessing the dynamic data sharing interface of the dynamic data sharing module.

[0059] Furthermore, by configuring directed acyclic graphs (DAGs), multi-business data from oil and gas pipelines can be correlated with multi-step business processes. This allows dynamic data to be processed during the flow of data, fulfilling additional data correlation requirements. This is because, during data loading, some complex dynamic business data requires data cleaning. For example, dynamic data related to leak monitoring, critical equipment and facility health monitoring, production metering, and data acquisition and monitoring systems often need to be correlated with equipment description data during synchronization for data filtering and supplementation. However, traditional dynamic data loading technologies suffer from complex synchronization configuration operations and limited DAG functionality, making them unable to handle complex dynamic business data processing, let alone achieve the expected data cleaning effect.

[0060] The dynamic data sharing module is specifically designed to address the issue of slow dynamic data synchronization. Utilizing distributed system design principles, it has developed multi-channel, multi-threaded, traceable, and directed acyclic graph (DAG) data synchronization capabilities. Multi-channel functionality leverages the computer's multi-core resources, allowing for multi-channel data processing through code manipulation. Multi-threading leverages the computer's thread resources, enabling multi-threaded data processing. Traceability is achieved by logging data processing in the background, providing a traceable and systematic tracking mechanism. The DAG, achieved through object-oriented programming, abstracts each step as an object entity, enabling DAG functionality through communication between objects. The distributed design allows for horizontal scaling of computer resources when single-machine resources are insufficient for rapid dynamic data synchronization. Machine resources are managed via public or private clouds, and additional resources can be added through the cloud when resources are insufficient, achieving horizontal scaling. Increasing the number of dynamic data synchronization threads or channels linearly expands data loading capacity to solve the problem.

[0061] The dynamic data synchronization module internally builds interfaces for interfacing with various business database protocols, replacing upper-layer applications and directly connecting to diverse heterogeneous data. Dynamic data such as oil and gas flow, tank levels, and pipeline inlet and outlet pressures are stored in relational databases. The implementation of multi-database business system access protocols in the dynamic data synchronization module allows upper-layer applications to avoid interfacing with heterogeneous data from multiple business data systems, thereby reducing data redundancy, streamlining design, and adhering to the software engineering principles of low coupling and high cohesion. This avoids data redundancy and bloated design in upper-layer applications.

[0062] After the dynamic data synchronization module component circumvents the data access protocol types of upper-layer applications connecting to multi-service database systems, this invention sets up a dynamic data sharing module component to simplify the connection steps for upper-layer applications. As the sole data sharing interface of this technology, the dynamic data sharing module component greatly facilitates the dynamic data access needs of upper-layer applications for multi-service data systems. Upper-layer applications only need to interact with the dynamic data sharing interface provided by this technology's dynamic data sharing module.

[0063] This invention's dynamic data storage module is primarily designed to address the limitations of single-machine resources in storing large volumes of dynamic data. Leveraging the design principles of distributed systems in the internet domain, the module offers high availability, distributed architecture, horizontal scalability, large capacity, and multi-type data processing capabilities. It overcomes the major drawbacks of traditional dynamic data loading technologies, such as the inability to horizontally scale machine resources, limited computing power, and significant storage space limitations. When data volume increases dramatically, additional machine resources can be added to increase storage space and improve computing power. Traditional dynamic data loading technologies suffer from declining performance due to the accumulation of historical data. The inherently distributed structure, dynamic partitioning, and isolation of hot and cold data in this invention's dynamic data storage module make storing massive amounts of historical data and loading hot dynamic data both easy and accessible.

[0064] Furthermore, traditional dynamic data loading technologies, while capable of adding indexes with the influx of massive amounts of data, suffer from slow loading speeds for upper-layer applications and time-consuming calls to business systems due to the increasing historical storage of such data. This technology's dynamic data storage module significantly improves dynamic data loading efficiency by implementing columnar storage, sharded storage, memory buffering, and dynamic partitioning, providing highly efficient services to upper-layer applications. Columnar storage breaks away from the traditional row-based storage model, arranging each field according to columns using a merge tree. Sharded storage utilizes distributed machine resources to distribute data across multiple machines. Memory buffering caches data, pre-loading data from disk into memory to improve access efficiency. Dynamic partitioning extracts commonalities in data, such as data related to a specific time point, allowing data to be partitioned into time-based directories for efficient access. Data of the same type is grouped into the same partition, further accelerating access efficiency. In columnar storage technology, traditional dynamic data loading techniques often return an entire row of data when loading dynamic data. However, upper-layer applications, in the case of multiple data items, only need to read data from a limited number of fields, not an entire row. Therefore, this method of reading entire rows of data in traditional dynamic data loading technology greatly increases the consumption of machine resource I / O ports and cannot optimize loading speed. Columnar storage, on the other hand, can return only the read and load of specified data items, reducing the reading and loading of unnecessary data items, lowering the consumption of machine resource I / O ports, and improving the loading speed of dynamic data.

[0065] In other words, the dynamic data loading technology of the present invention sets up three functional components: a dynamic data synchronization module, a dynamic data sharing module, and a dynamic data storage module, which are used to realize the functions of dynamic data synchronization, dynamic data sharing, and dynamic data storage, respectively.

[0066] Taking crude oil flow data as an example, the dynamic data loading or transmission process is as follows:

[0067] S11. Access the visual operation interface of the dynamic data synchronization module. This interface allows for the configuration of dynamic data synchronization, enabling the dynamic data synchronization function. Add a dynamic data source configuration for crude oil flow in the interface, selecting the specific data access protocol type.

[0068] S12. After connecting the crude oil flow dynamic data source and the dynamic data synchronization module component, select the specific fields to be synchronized. Then, establish the corresponding data model in the dynamic data storage module, which is the structure of the data table in the database. Store the data synchronized from the crude oil flow dynamic data source.

[0069] The dynamic data synchronization module provides interfaces for multiple business database data access protocols to the outside world, synchronizing dynamic crude oil flow data, and internally storing the dynamic crude oil flow data in the storage space provided by the dynamic data storage module.

[0070] Once the dynamic data source for crude oil flow is connected to the dynamic data synchronization module, the appropriate number of channels, concurrency, and threads are configured based on the size of the dynamic data volume and the data generation speed. Configuring these parameters involves manipulating the computer's underlying processes and threads through code. Generally, for data volumes of 50 mg / s, the number of channels is configured to 1 and the number of threads to 1; for 100 mg / s, the number of channels and threads to 2; and for 500 mg / s, the number of channels and threads to 3. However, more is not always better; appropriateness is paramount. Then, the time scheduling is set in the visual dynamic data synchronization module's operation interface to synchronize the dynamic crude oil flow data in real time, ultimately resolving to the dynamic data storage module component.

[0071] The dynamic data synchronization module synchronizes dynamic data in real time to the dynamic data storage module component, providing storage assurance for upper-layer applications to share dynamic data. Here, the crude oil flow dynamic data will be synchronized in real time to the corresponding data model in the dynamic data storage module component. Using dynamic partitioning technology, the crude oil flow dynamic data is partitioned according to a time field, enabling upper-layer applications to efficiently load dynamic data. The partitioning method is based on directory division according to data characteristics, with data of the same type grouped into the same directory.

[0072] S13. Upper-layer applications load crude oil flow dynamic data in real time by accessing the dynamic data sharing interface of the dynamic data sharing module. The dynamic data sharing module component provides a consistent data sharing interface externally and internally reads the crude oil flow dynamic data stored in the dynamic data storage module component.

[0073] The dynamic data sharing module, serving as the dynamic data sharing service function of this technology, allows upper-layer applications to avoid interfacing with various heterogeneous data access protocols. They only need to interface with the dynamic data sharing module to access real-time synchronized massive amounts of dynamic data. The dynamic data sharing module provides access interfaces for programs written in languages ​​such as Java, C#, C / C++, Python, and Javascript, meeting the dynamic data loading needs of various services and multiple shared locations.

[0074] The dynamic data transmission and loading system designed in this embodiment of the invention can solve the problem of unified data collection from multi-source heterogeneous business systems with high timeliness, large data volume and diverse structure by setting up a dynamic data loading middleware. It can also simplify the design of upper-layer applications, alleviate the dynamic loading pressure of upper-layer applications, and facilitate the later maintenance of upper-layer applications.

[0075] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the concept and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for dynamic transmission and loading of multi-service data in oil and gas pipelines, characterized in that, Includes the following steps: Based on the size and data generation speed of the multi-service data of the oil and gas pipeline, the number of data transmission channels, the number of concurrency, and the number of threads are automatically configured; Based on the automatically configured number of data transmission channels, the number of concurrent connections, and the number of threads, the multi-service data of the oil and gas pipeline is collected or synchronized to obtain dynamic synchronization data; The dynamically synchronized data is stored in the dynamic data storage module; The system reads the dynamic synchronization data from the dynamic data storage module and provides a dynamic data sharing interface to external systems. The upper-layer application loads the dynamically synchronized data by accessing the dynamic data sharing interface, thereby realizing the data processing of the multi-service data of the oil and gas pipeline. Specifically, the data processing involves data transmission or loading, which collects or synchronizes the multi-service data of the oil and gas pipeline based on the automatically configured number of data transmission channels, the number of concurrent connections, and the number of threads to obtain dynamically synchronized data. This includes the following steps: The oil and gas pipeline multi-service data is collected or synchronized according to the automatically configured number of data transmission channels, the number of concurrent connections, and the number of threads. A directed acyclic graph data topology is established, and the association structure of the multi-business data of the oil and gas pipeline is established using the directed acyclic graph data topology to obtain the dynamic synchronization data. Based on the size and data generation speed of the multi-service data from the oil and gas pipeline, the number of data transmission channels, concurrency, and threads are automatically configured, including the following steps: Based on the size and data generation speed of the multi-service data of the oil and gas pipeline, the number of data transmission channels, concurrency, and threads of each node in the distributed system are automatically configured according to the distributed system design model. Access the visual operation interface of the dynamic data synchronization module. On this interface, you can configure the dynamic data synchronization and realize the dynamic data synchronization function. Add the configuration of the dynamic data source of crude oil flow in the interface and select the specific data access protocol type. After the dynamic data source of crude oil flow is connected to the dynamic data synchronization module component, select the specific fields that need to be synchronized, and then establish the structure of the corresponding data table in the dynamic data storage module to store the data synchronized from the dynamic data source of crude oil flow. The dynamic data synchronization module provides interfaces for multiple business database data access protocol types to synchronize crude oil flow dynamic data, and internally stores the crude oil flow dynamic data in the storage space provided by the dynamic data storage module. Once the dynamic data source of crude oil flow is connected to the dynamic data synchronization module, the number of channels, concurrency, and threads are configured reasonably based on the size of the dynamic data and the data generation speed. Configuring the number of channels, concurrency, and threads is achieved by manipulating the underlying processes and threads of the computer through code.

2. The method for dynamic transmission and loading of multi-service data for oil and gas pipelines according to claim 1, characterized in that, During the process of collecting or synchronizing the multi-service data of the oil and gas pipeline, corresponding data processing logs are also generated.

3. The method for dynamic transmission and loading of multi-service data for oil and gas pipelines according to claim 1, characterized in that, The process of storing the dynamically synchronized data into the dynamic data storage module includes the following steps: Using a dynamic partitioning method, the dynamic synchronization data is partitioned according to the time field, and the partitioned dynamic synchronization data is stored in the dynamic data storage module.

4. The method for dynamic transmission and loading of multi-service data for oil and gas pipelines according to claim 3, characterized in that, The dynamic synchronization data after partitioning is stored in the dynamic data storage module, including the following steps: The dynamically synchronized data after partitioning is stored in the dynamic data storage module in a columnar storage format.

5. The method for dynamic transmission and loading of multi-service data for oil and gas pipelines according to claim 2, characterized in that, The process of storing the dynamically synchronized data into the dynamic data storage module further includes the following steps: The dynamically synchronized data and the data processing logs are stored in the dynamic data storage module.

6. The method for dynamic transmission and loading of multi-service data for oil and gas pipelines according to claim 1, characterized in that, The process of reading the dynamically synchronized data from the dynamic data storage module and providing a dynamic data sharing interface includes the following steps: The dynamic synchronization data is read from the dynamic data storage module, and the dynamic data sharing interface is provided to the outside world based on the principle of unified interface.

7. A dynamic transmission and loading system for multi-service data in oil and gas pipelines, characterized in that: include, The dynamic data synchronization module is used to automatically configure the number of data transmission channels, the number of concurrent connections, and the number of threads based on the size and data generation speed of the multi-service data of the oil and gas pipeline; and to collect or synchronize the multi-service data of the oil and gas pipeline based on the automatically configured number of data transmission channels, the number of concurrent connections, and the number of threads to obtain dynamic synchronization data. A dynamic data storage module is used to store the dynamically synchronized data; The dynamic data sharing module is used to read the dynamically synchronized data from the dynamic data storage module and provide a dynamic data sharing interface to the outside world; wherein, the upper-layer application loads the dynamically synchronized data by accessing the dynamic data sharing interface to realize the data processing of multi-business data of the oil and gas pipeline; wherein, the data processing specifically refers to data transmission or data loading. The process of collecting or synchronizing multi-service data from oil and gas pipelines based on the automatically configured number of data transmission channels, the number of concurrent connections, and the number of threads to obtain dynamically synchronized data includes the following steps: The oil and gas pipeline multi-service data is collected or synchronized according to the automatically configured number of data transmission channels, the number of concurrent connections, and the number of threads. A directed acyclic graph data topology is established, and the association structure of the multi-business data of the oil and gas pipeline is established using the directed acyclic graph data topology to obtain the dynamic synchronization data. Based on the size and data generation speed of the multi-service data from the oil and gas pipeline, the number of data transmission channels, concurrency, and threads are automatically configured, including the following steps: Based on the size and data generation speed of the multi-service data of the oil and gas pipeline, the number of data transmission channels, concurrency, and threads of each node in the distributed system are automatically configured according to the distributed system design model. Access the visual operation interface of the dynamic data synchronization module. On this interface, you can configure the dynamic data synchronization and realize the dynamic data synchronization function. Add the configuration of the dynamic data source of crude oil flow in the interface and select the specific data access protocol type. After the dynamic data source of crude oil flow is connected to the dynamic data synchronization module component, select the specific fields that need to be synchronized, and then establish the structure of the corresponding data table in the dynamic data storage module to store the data synchronized from the dynamic data source of crude oil flow. The dynamic data synchronization module provides interfaces for multiple business database data access protocol types to synchronize crude oil flow dynamic data, and internally stores the crude oil flow dynamic data in the storage space provided by the dynamic data storage module. Once the dynamic data source of crude oil flow is connected to the dynamic data synchronization module, the number of channels, concurrency, and threads are configured reasonably based on the size of the dynamic data and the data generation speed. Configuring the number of channels, concurrency, and threads is achieved by manipulating the underlying processes and threads of the computer through code.

Images