A data synchronization method for a two-level data center of a coal mine

Abstract

The application discloses a kind of data synchronization methods of coal mine two-stage data center, comprising the following steps: S1, by data acquisition adapter, coal mine data is converged to coal mine end data center;S2, while writing coal mine data into coal mine end data center, after the coal mine data is parsed, it is mirrored from first business data message queue cluster to second business data message queue;S3, from second business data message queue, mirror after the coal mine data is obtained, and is written into company end data center by second warehousing service;S4;Coal mine related data generated by company end is mirrored from second business data message queue cluster to first business data message queue;S5, from first business data message queue, the coal mine related data is obtained, and is written into coal mine end data center by first warehousing service.The scheme keeps the data consistency of two data centers, improves the real-time performance of data synchronization, meets the efficiency requirement of coal mine actual data storage and application.

Description

Technical Field

[0001] This invention belongs to the field of coal mine data interaction and synchronization, and relates to a data synchronization method for two-level data centers in a coal mine. Background Technology

[0002] In coal mine mining and management, it is typically necessary to establish data centers at both the mine and company levels. The mine-side data center primarily aggregates all data related to mine safety, production, scheduling, and operations, while the company-side data center aggregates data from various mine-side data centers as well as the company's own data. Therefore, ensuring data consistency between the two data centers becomes a critical challenge.

[0003] In the coal mining industry, the most common data center synchronization refers to the synchronization of the database between the coal mine and the company. This typically employs a combination of data read / write separation and data log synchronization. This method usually separates read and write operations between the two data centers, deploying two database servers—one for read operations and the other for write operations. Simultaneously, the write operation logs are parsed and synchronized between the two data centers. However, this approach is generally suitable for management systems with low data generation frequency and small data volumes, and is unsuitable for the specific industrial scenarios of coal mines. Coal mines generate large amounts of environmental monitoring and equipment operating data frequently during monitoring and production process control. This data needs to be collected and stored quickly. Given the high frequency and large volume of data, log synchronization cannot meet the efficiency requirements of actual data storage and application in coal mines. Summary of the Invention

[0004] In view of this, the purpose of this invention is to provide a data synchronization method for two-level data centers in coal mines, which achieves data consistency between the two data centers through bidirectional data synchronization, connects the data channels between the production execution layer and the control layer, improves the real-time performance of data synchronization, meets the efficiency requirements of actual data storage and application in coal mines, and provides data support for the intelligent construction of coal mines.

[0005] To achieve the above objectives, the present invention provides the following technical solution:

[0006] A data synchronization method for a two-level data center in a coal mine, based on a coal mine end and a company end, involves deploying a data acquisition adapter, a data exchange service, a data parser, a raw data message queue, a first business data message queue, a first data entry service, and a coal mine-side data center at the coal mine end, and deploying a second business data message queue, a second data entry service, and a company-side data center at the company end, including the following steps:

[0007] S1. Aggregate coal mine data to the coal mine data center;

[0008] Coal mine data aggregation specifically includes:

[0009] S11. Adapt the data sources of each subsystem of the coal mine to the data acquisition adapter and collect them uniformly; the data sources include monitoring data, production control data and operation and management data.

[0010] S12. After combining the data set from the data source with the data pushed by the third-party platform to form coal mine data, back it up locally as a text file to obtain the first backup file, and submit it to the data exchange service.

[0011] S13. Perform preliminary data verification on the coal mine data through the data exchange service; if the preliminary data verification is successful, push the coal mine data to the original data message queue; if the verification fails, push it to the non-compliant record topic in the original data message queue.

[0012] S14. Obtain data from the original data message queue, parse the coal mine data through the data parser, push the parsed coal mine data to the first business data message queue, and write it into the coal mine data center through the first data entry service.

[0013] S2. While writing the parsed coal mine data to the coal mine data center, the parsed coal mine data is mirrored from the first business data message queue cluster to the second business data message queue.

[0014] S3. Obtain the mirrored coal mine data from the second business data message queue and write it to the company's data center through the second data entry service.

[0015] S4. Mirror the coal mine-related data generated by the company from the second business data message queue cluster to the first business data message queue.

[0016] S5. Obtain the coal mine-related data from the first business data message queue and write it into the coal mine-side data center through the first data entry service.

[0017] Furthermore, in S12, the first business data message queue cluster is divided into multiple message topics according to the different characteristics of coal mine data. The message topics include definition data topics, real-time data topics, and other data topics.

[0018] This solution assigns each data subject to transmit different types of data, enabling the separate transmission of data with different characteristics. This allows for the identification of transmission status based on data characteristics, facilitating the detection of mistransmissions and omissions.

[0019] Furthermore, the message topics are sorted in descending order of priority as follows: definition data topic > real-time data topic > other data topics.

[0020] This solution can improve the timing of data transmission, determine whether the transmission is carried out in the predetermined time sequence, thus facilitating the judgment of whether data transmission has failed, and quickly retransmitting necessary data after a failure, avoiding the situation where necessary data is missed.

[0021] Furthermore, in S12, if data submission fails, a submission failure policy is executed:

[0022] 1) If the data interaction service transmission interface is in normal status, resubmit immediately once;

[0023] 2) If the data interaction service transmission interface is unavailable and the data cannot be discarded, mark the first backup file as a failed submission and wait for resubmission;

[0024] 3) If the data interaction service transmission interface is unavailable and the data can be discarded, then the recording will end and resubmission will be stopped.

[0025] Furthermore, in S2 and S4, the cluster mirroring process is as follows:

[0026] Step 1: The data mirroring module operates in a producer-consumer model, allocating a thread to each consumer. Consumers read data from different message topics in the source cluster and then send the data to the target cluster through a common producer.

[0027] Step 2: Every preset period, the consumer notifies the producer to send data to the target cluster and waits for confirmation from the target cluster.

[0028] Step 3: The consumer notifies the source cluster to submit the corresponding offset.

[0029] In S2, the first business data message queue is the source cluster, and the second business data message queue is the target cluster; in S5, the second business data message queue is the source cluster, and the first business data message queue is the target cluster.

[0030] Before the source cluster submits the offset, the message queue confirms the message, which ensures that no data is lost. Moreover, if the data mirroring process crashes, there will be at most a short period of duplicate data (preset period), which reduces the amount of redundant data that may be generated during the data synchronization process.

[0031] Furthermore, the data parsing process includes:

[0032] S141. Obtain non-standard data and standard data from the original data message queue, back them up to a local data file, and obtain a second backup file;

[0033] S142. Obtain non-standard data verification rules, verify non-standard data, and classify non-standard data into non-compliant non-standard data and compliant non-standard data; push non-compliant non-standard data to the first business data message queue; obtain data conversion rules, and convert compliant non-standard data into standard data;

[0034] S143. Obtain standard data verification rules, verify the standard data, and divide the standard data into non-compliant standard data and compliant standard data; push the non-compliant standard data to the first business data message queue, obtain data conversion rules, and convert the compliant standard data into business data;

[0035] S144. Perform event analysis on business data, including analysis of measurement point changes, anomaly records, and hierarchical early warnings.

[0036] S145. Push the business data to the first business data message queue.

[0037] Furthermore, in the first and second data entry services, the first and second business data message queues are respectively used as business data message queues, and the data entry process includes:

[0038] 1) Consume business data in the message queue, back it up to a local data file, and obtain a third backup file;

[0039] 2) Load the storage strategy corresponding to the message topic and assemble the data into a DataTable storage structure;

[0040] 3) Write the data to the target database according to the storage policy. If the writing is successful, submit the message consumption offset in the queue; otherwise, execute the storage failure policy.

[0041] Definitions:

[0042] Consumption: A specific term in the consumer-producer work model, referring to a data processing process; consuming business data means acquiring business data.

[0043] DataTable: A data table is a data storage structure.

[0044] Furthermore, the data entry failure strategy includes:

[0045] When the target database is Redis: if the number of failures is less than 3, try to re-insert the data; if the number of failures reaches 3, commit the message consumption offset, end the insertion process and record the result.

[0046] When the target database is HBase: if the number of failures is less than 3, try to re-insert the data; if the number of failures reaches 3, end the data insertion process and record the failure.

[0047] When the target database is a relational database: if the data insertion fails due to resource usage or access timeout, re-insert the data; if the data insertion fails due to an incorrect data format and the insertion method is batch insertion, change to writing data row by row and re-insert the data; if the data insertion fails due to an incorrect data format and the insertion method is writing data row by row, submit the message consumption offset, end the insertion process and record the result.

[0048] The beneficial effects of this invention are as follows:

[0049] This solution utilizes data acquisition and transmission, data communication, data mirroring, and software engineering technologies to synchronously write monitoring and production operation data related to the coal mine, as well as data related to the company and the coal mine, into two-tier data centers of the coal mine and the company. By employing data message queues, data mirroring, and data synchronization parsing, it achieves data consistency between the two-tier data centers of the coal mine and the company, improves the real-time performance of data synchronization, meets the efficiency requirements of actual data storage and application in the coal mine, and achieves the goal of data transparency and integration between the coal mine and the company, providing data support for the intelligent construction of coal mines.

[0050] Other advantages, objectives, and features of the invention will be set forth in part in the description which follows, and in part will be apparent to those skilled in the art from the following examination, or may be learned from practice of the invention. The objectives and other advantages of the invention can be realized and obtained through the following description. Attached Figure Description

[0051] To make the objectives, technical solutions, and advantages of the present invention clearer, the preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings, wherein:

[0052] Figure 1 This is a diagram of the synchronization architecture of a two-level data center in a coal mine according to an embodiment of the present invention.

[0053] Figure 2 This is a schematic diagram illustrating the data synchronization principle of a two-level data center in a coal mine, according to an embodiment of the present invention. Detailed Implementation

[0054] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0055] The accompanying drawings are for illustrative purposes only and are schematic diagrams, not actual pictures. They should not be construed as limiting the invention. To better illustrate the embodiments of the invention, some parts in the drawings may be omitted, enlarged, or reduced, and do not represent the actual product dimensions. It is understandable to those skilled in the art that some well-known structures and their descriptions may be omitted in the drawings.

[0056] In the accompanying drawings of the embodiments of the present invention, the same or similar reference numerals correspond to the same or similar components. In the description of the present invention, it should be understood that if terms such as "upper," "lower," "left," "right," "front," and "rear" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, they are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, the terms used to describe positional relationships in the drawings are only for illustrative purposes and should not be construed as limiting the present invention. For those skilled in the art, the specific meaning of the above terms can be understood according to the specific circumstances.

[0057] Please see Figures 1-2 This is a data synchronization method for a two-level data center in a coal mine, based on a coal mine end and a company end. The coal mine end has a coal mine data center, and the company end has a company data center. A data acquisition adapter, a data exchange service, and a first business data message queue (i.e., ...) are deployed at the coal mine end. Figure 1 On the coal mine side, a business data message queue, a data parser, a coal mine-side data entry service, a coal mine-side data analysis service, and a coal mine management and control platform are deployed. A second business data message queue (i.e., ...) is deployed on the company side. Figure 1 The business data message queue on the company side, the company-side data analysis service, the company-side warehousing service, and the company management platform include the following steps:

[0058] S1. Aggregate coal mine data to the coal mine data center;

[0059] Coal mine data aggregation specifically includes:

[0060] S11. Adapt the data sources of each subsystem of the coal mine to the data acquisition adapter and collect them uniformly; the data sources include monitoring data, production control data and operation and management data.

[0061] S12. After combining the data set from the data source with the data pushed by the third-party platform to form coal mine data, back it up locally as a text file to obtain the first backup file, and submit it to the data exchange service.

[0062] S13. Perform preliminary data verification on the coal mine data through the data exchange service; if the preliminary data verification is successful, push the coal mine data to the original data message queue; if the verification fails, push it to the non-compliant record topic in the original data message queue.

[0063] S14. Obtain data from the original data message queue, parse the coal mine data through the data parser, push the parsed coal mine data to the first business data message queue, and write it into the coal mine data center through the first data entry service.

[0064] S2. While writing the parsed coal mine data to the coal mine data center, the parsed coal mine data is mirrored from the first business data message queue cluster to the second business data message queue.

[0065] S3. Obtain the mirrored coal mine data from the second business data message queue and write it to the company's data center through the second data entry service.

[0066] S4. Mirror the coal mine-related data generated by the company from the second business data message queue cluster to the first business data message queue.

[0067] S5. Obtain the coal mine-related data from the first business data message queue and write it into the coal mine-side data center through the first data entry service.

[0068] like Figure 1 As shown, there are four sources of data at the coal mine end, three of which are from various subsystems at the coal mine end, and the other is from third-party platforms: (1) Data generated by monitoring and control subsystems. For example, coal mines have installed monitoring and control subsystems such as safety monitoring, personnel positioning, emergency broadcasting, hydrological monitoring, and communication. These subsystems collect various monitoring and control data in real time to the corresponding subsystems through sensors and other equipment deployed underground or on the ground. (2) Data generated by production control subsystems. Large electromechanical equipment is installed in the production links of coal mines such as mining, tunneling, machinery, transportation, and ventilation. Generally, some control subsystems are formed based on these equipment, such as coal mining machine monitoring, tunneling machine monitoring, main ventilation monitoring, main transportation monitoring, main drainage pump monitoring, compressed air nitrogen production, and other systems. (3) Data generated by operation and management subsystems. In addition to the aforementioned monitoring and control and production process control, there are also separate subsystems for coal mine safety management and production scheduling, such as hidden danger management, risk point control, production scheduling, production indicators, and accident management subsystems. (4) Data pushed by third-party platforms to the coal mine data center.

[0069] These subsystems are typically deployed and run independently, each with its own data repository, creating a series of information silos. This solution uses a data acquisition adapter to uniformly collect and adapt the data from these subsystems, then submits it to a data exchange service. This data, along with data pushed to the coal mine data center by a third-party platform, is parsed together and ultimately all the data is integrated into the unified data center of the coal mine, improving the efficiency of data storage and application.

[0070] The company's data comes from two sources: (1) various types of data submitted by the coal mines; and (2) business data related to the coal mines generated by the company's management platform, i.e., coal mine-related data. Among these, the coal mine-related data needs to be synchronized to the coal mine data center.

[0071] The core element in achieving data synchronization between the coal mine and company-level data centers is the data synchronization between the business data message queue clusters at both ends. Because the data message queues at both ends are deployed in a cluster, they provide the physical data channel for data synchronization between the two data centers. Based on different data characteristics, the coal mine's data message queue cluster is divided into multiple data topics, such as defined data topics, real-time data topics, and other data topics (e.g., historical data topics), with each data topic responsible for data transmission of different characteristic types. The company's data message queue cluster (i.e., the second business data passive queue) has the same topic configuration as the coal mine's data message queue cluster (i.e., the first business data message queue).

[0072] The entire data synchronization is accomplished through the data mirroring module, such as... Figure 2 As shown, the specific process of cluster mirroring in S2 and S4 above is as follows:

[0073] Step 1: The data mirroring module operates in a producer-consumer model, allocating a thread to each consumer. Consumers read data from different message topics in the source cluster and then send the data to the target cluster through a common producer.

[0074] Step 2: Every preset period, the consumer notifies the producer to send data to the target cluster and waits for confirmation from the target cluster; in this embodiment, the preset period is 30 seconds.

[0075] Step 3: The consumer notifies the source cluster to submit the corresponding offset.

[0076] In S2, the first business data message queue is the source cluster, and the second business data message queue is the target cluster; in S5, the second business data message queue is the source cluster, and the first business data message queue is the target cluster. This ensures that no data is lost (the message queue acknowledges the message before the source cluster commits the offset), and if the data mirroring process crashes, at most only short-term (the interval of periodic notifications, 30 seconds in this embodiment) duplicate data will occur.

[0077] The specific principles and methods of each functional module deployed at the coal mine and company ends in this solution are as follows:

[0078] 1. Data acquisition adapter

[0079] The data acquisition adapter collects data from various subsystems in the coal mine and submits it to the data exchange service. The data exchange service performs preliminary data verification and then writes the data to the raw data message queue. The data acquisition adapter contains three key modules: data acquisition adaptation, data submission, and data backup.

[0080] (1) Data acquisition adaptation

[0081] Coal mine subsystems provide data in various ways, including text files (TXT), databases, web APIs, OPC DA, OPC UA, and EIP. Data acquisition adapters need to adapt to these data sources, acquire data from the data sources at a specified acquisition frequency, and convert it into an in-memory dataset. Data acquisition adapters are compatible with multiple data protocols, including text files (TXT), databases, web APIs, OPC DA, OPC UA, and EIP.

[0082] (2) Data Submission

[0083] After obtaining the data set from the data source, the system first retrieves the data from the data source, backs it up to a local data file to obtain the first backup file, and adds it to the pending submission queue. Then, it reads the data file to be retransmitted from the first backup file and adds it to the pending submission queue. The data is divided into definition data, real-time data, and other data, each corresponding to a topic partition in the first business data message queue. Different data is transmitted through different topic partitions. When submitting data, the priority is defined data > real-time data > other data. This approach distinguishes the timing of data transmission and confirms whether data must be submitted in sequence. For example, if definition data and real-time data are submitted in sequence but the actual submission sequence differs, it indicates a mistransmission or omission, requiring resubmission to ensure correct submission. Simultaneously, because the message topics are clearly divided according to the characteristics of coal mine data, when data submission fails, it quickly identifies which part of the data has been submitted incorrectly or incompletely, and determines whether data must be submitted based on its importance. Less important data can be abandoned and retransmitted, thereby improving data submission efficiency.

[0084] The specific data submission failure strategies include:

[0085] The data interaction service transmission interface is in normal status. Retransmit once immediately.

[0086] The data interaction service transmission interface is unavailable, and the data must be committed. The first backup file is marked as having failed to commit and is awaiting retransmission.

[0087] The data interaction service transmission interface is unavailable and the data can be discarded, ending the record.

[0088] (3) Submit data backup

[0089] The data backup module has the following functions:

[0090] 1) Back up data to local files;

[0091] 2) Supports file tagging functionality to identify files that failed to be processed;

[0092] 3) Supports querying and reading backup files;

[0093] 4) Expired data files are automatically deleted;

[0094] 5) Supports compressed file storage.

[0095] 2. Data exchange service

[0096] The data exchange service runs as a standalone application, supports distributed deployment, and provides a web API-based data interaction interface. It is responsible for receiving data pushed by the data acquisition adapter and third-party platforms and transmitting it to the message queue, as well as responding to business data requests from third-party platforms. It includes functions such as authorization verification and data transmission.

[0097] (1) Permission verification: Verify user identity information, record login time, and return the authentication token of the data interaction interface (a JSON string containing user information and login time, encoded in base64).

[0098] (2) Data transmission: The data exchange service first receives data pushed from the data acquisition adapter and third-party platforms, and backs it up to local files; then it performs preliminary data verification. If the verification is successful, the data is pushed to the original data message queue; otherwise, the data is pushed to the non-compliant record topic in the original data message queue. In addition, it also has a data push frequency control function, which records the data push time according to the dimensions of coal mine, business system, and data type, and controls the data push frequency according to different data types.

[0099] 3. Data parser

[0100] The data parser runs as a standalone application and includes modules for parsing strategy management, data parsing, and parsed data backup. Its core functions are the conversion of non-standard data and the parsing of standard data. The processing flow includes defining a data model, consuming coal mine data from the raw data message queue, performing data verification, data conversion, data parsing, and event analysis, and then pushing the processed business data to the message queue to await entry into the database.

[0101] (1) Analysis Strategy Management

[0102] The data parser first loads the definition information such as data parsing strategy, data validation rules and data transformation methods from the configuration file for data parsing.

[0103] (2) Data Analysis

[0104] Data parsing process;

[0105] S141. Obtain non-standard data and standard data from the original data message queue, back them up to a local data file, and obtain a second backup file;

[0106] S142. Obtain non-standard data verification rules, verify non-standard data, and classify non-standard data into non-compliant non-standard data and compliant non-standard data; push non-compliant non-standard data to the first business data message queue; obtain data conversion rules, and convert compliant non-standard data into standard data;

[0107] S143. Obtain standard data verification rules, verify the standard data, and divide the standard data into non-compliant standard data and compliant standard data; push the non-compliant standard data to the first business data message queue, obtain data conversion rules, and convert the compliant standard data into business data;

[0108] S144. Perform event analysis on business data, such as analysis of measurement point changes, anomaly records, and hierarchical early warnings;

[0109] S145. Write real-time hot data to the Redis in-memory database to increase the reliability of cached data; push business data to the first business data message queue, waiting for the data entry service to write it to the data center.

[0110] 4. Warehousing service

[0111] The inbound service runs as a standalone application, comprising two main modules: inbound policy management and data inbound. It supports distributed deployment. Its core function is to consume business data from the business data message queue, assemble it into a data storage structure, and ensure accurate data inbound based on the data storage policy and the characteristics of the target database.

[0112] (1) Warehouse entry strategy management

[0113] The inbound service reads the inbound storage strategy from the configuration file and synchronizes the inbound strategies for various message topics, including the target database table structure, inbound method (SQL), row key format (Redis, HBase), and real-time data expiration time. In addition, it maintains the database (table) structure, creates tables, and upgrades table structures.

[0114] (2) Data import

[0115] Data import includes the following processes:

[0116] 1) Consume business data from the message queue, back it up to a local data file, and obtain a third backup file;

[0117] 2) Load the storage strategy related to the message topic and assemble the data into a DataTable storage structure;

[0118] 3) According to the storage strategy, write to the target database in the specified way. If the write is successful, submit the message consumption offset in the queue; otherwise, execute the relevant failure strategy.

[0119] The database import failure strategy is divided into the following three categories depending on the target database:

[0120] 1) If the target database is Redis: If the number of failures is less than 3, try to re-insert the data; if the number of failures reaches 3, commit the message consumption offset, end the insertion process and record the result.

[0121] 2) If the target database is HBase: If the number of failures is less than 3, try to re-insert the data; if the number of failures reaches 3, end the data insertion process and record the failure.

[0122] 3) If the target database is a relational database: If the data insertion fails due to resource usage or access timeout, re-insert the data; if the data insertion fails due to an incorrect data format and the data insertion method is batch insertion, change to writing data one record at a time and re-insert the data; if the data insertion fails due to an incorrect data format and the data insertion method is writing data one record at a time, submit the message consumption offset, end the data insertion process and record the result.

[0123] 5. Data Analysis Services

[0124] The data analytics service exists as a software module, available for registration and use by various applications. Its primary function is to perform secondary statistical analysis and data mining on data in the data center, for use by the coal mine management platform. The data analytics service processing flow includes loading a statistical analysis model, retrieving data from the target database or memory, performing data analysis, and then writing the results to the target database or a message queue for pending business data.

[0125] 6. Data Center

[0126] Data centers primarily store three types of data: real-time data, short-to-medium term data, and historical sampled data. Typically, Redis in-memory database, relational database, and HBase are used to store these three types of data respectively. Redis in-memory database is used to establish a real-time data cache, enabling efficient access to frequently accessed data. Relational database performance metrics are used for storage, employing methods such as database table partitioning. HBase, a column-oriented storage database, is used to store historical sampled data, supporting historical data backtracking.

[0127] 7. Coal Mine Management and Control Platform

[0128] Typically, coal mine data centers and coal mine management platforms are deployed and used in conjunction with each other. The coal mine management platform visualizes the data from the coal mine data center, performs related business functions, and saves the data generated by the platform to the data center.

[0129] 8. Data Message Queue

[0130] Data message queues are divided into two main categories: raw data message queues and business data message queues. Raw data message queues serve as an intermediate step in data parsing, providing support for the parsing process. Business data message queues are further divided into a primary business data message queue at the coal mine level and a secondary business data message queue at the company level. The core component for achieving data synchronization between the coal mine and company-level data centers is the business data message queue. The primary business data message queue at the coal mine level is deployed in a cluster, providing the physical data channel for data synchronization between the two data centers.

[0131] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A data synchronization method for a two-level data center in a coal mine, characterized in that: Based on the coal mine end and the company end, the system deploys a data acquisition adapter, data exchange service, data parser, raw data message queue, first business data message queue, first warehousing service, and coal mine end data center at the coal mine end, and deploys a second business data message queue, second warehousing service, and company end data center at the company end, including the following steps: S1. Aggregate coal mine data to the coal mine data center; Coal mine data aggregation specifically includes: S11. Adapt the data sources of each subsystem of the coal mine to the data acquisition adapter and collect them uniformly; the data sources include monitoring data, production control data and operation and management data. S12. After combining the data set from the data source with the data pushed by the third-party platform to form coal mine data, back it up locally as a text file to obtain the first backup file, and submit it to the data exchange service. S13. Perform preliminary data verification on the coal mine data through the data exchange service; if the preliminary data verification is successful, push the coal mine data to the original data message queue; if the verification fails, push it to the non-compliant record topic in the original data message queue. S14. Obtain data from the original data message queue, parse the coal mine data through the data parser, push the parsed coal mine data to the first business data message queue, and write it into the coal mine data center through the first data entry service. S2. While writing the parsed coal mine data to the coal mine data center, the parsed coal mine data is mirrored from the first business data message queue cluster to the second business data message queue. S3. Obtain the mirrored coal mine data from the second business data message queue and write it to the company's data center through the second data entry service. S4. Mirror the coal mine-related data generated by the company from the second business data message queue cluster to the first business data message queue. S5. Obtain the coal mine-related data from the first business data message queue and write it into the coal mine data center through the first data entry service. In steps S2 and S4, the cluster mirroring process is as follows: Step 1: The data mirroring module operates in a producer-consumer model, allocating a thread to each consumer. Consumers read data from different message topics in the source cluster and then send the data to the target cluster through a common producer. Step 2: Every preset period, the consumer notifies the producer to send data to the target cluster and waits for confirmation from the target cluster. Step 3: The consumer notifies the source cluster to submit the corresponding offset.

2. The data synchronization method for a two-level data center in a coal mine according to claim 1, characterized in that: In step S12, the first business data message queue is divided into multiple message topics according to the different characteristics of coal mine data. The message topics include definition data topics, real-time data topics, and other data topics.

3. The data synchronization method for a two-level data center in a coal mine according to claim 2, characterized in that: The message topics are sorted by priority from high to low as follows: definition data topic > real-time data topic > other data topics.

4. A data synchronization method for a two-level data center in a coal mine according to claim 1 or 3, characterized in that: In step S12, if data submission fails, the submission failure policy is executed: 1) If the data interaction service transmission interface is in normal status, resubmit immediately once; 2) If the data interaction service transmission interface is unavailable and the data cannot be discarded, mark the first backup file as a failed submission and wait for resubmission; 3) If the data interaction service transmission interface is unavailable and the data can be discarded, then the recording will end and resubmission will be stopped.

5. The data synchronization method for a two-level data center in a coal mine according to claim 1, characterized in that: The data parsing process includes: S141. Obtain non-standard data and standard data from the original data message queue, back them up to a local data file, and obtain a second backup file; S142. Obtain non-standard data verification rules, verify non-standard data, and classify non-standard data into non-compliant non-standard data and compliant non-standard data; push non-compliant non-standard data to the first business data message queue; obtain data conversion rules, and convert compliant non-standard data into standard data; S143. Obtain standard data verification rules, verify the standard data, and divide the standard data into non-compliant standard data and compliant standard data; push the non-compliant standard data to the first business data message queue, obtain data conversion rules, and convert the compliant standard data into business data; S144. Perform event analysis on business data, including monitoring point definition changes, anomaly records, and hierarchical early warning analysis. S145. Push the business data to the first business data message queue.

6. The data synchronization method for a two-level data center in a coal mine according to claim 5, characterized in that: In the first and second data ingestion services, the first and second business data message queues are used as business data message queues, respectively. The data ingestion process includes: 1) Consume business data in the message queue, back it up to a local data file, and obtain a third backup file; 2) Load the storage strategy corresponding to the message topic and assemble the data into a DataTable storage structure; 3) Write the data to the target database according to the storage policy. If the writing is successful, submit the message consumption offset in the queue; otherwise, execute the storage failure policy.

7. The data synchronization method for a two-level data center in a coal mine according to claim 6, characterized in that: The inbound failure strategy includes: When the target database is Redis: if the number of failures is less than 3, try to re-insert the data; if the number of failures reaches 3, commit the message consumption offset, end the insertion process and record the result. When the target database is HBase: if the number of failures is less than 3, try to re-insert the data; if the number of failures reaches 3, end the data insertion process and record the failure. When the target database is a relational database: if the data insertion fails due to resource usage or access timeout, re-insert the data; if the data insertion fails due to an incorrect data format and the insertion method is batch insertion, change to writing data row by row and re-insert the data; if the data insertion fails due to an incorrect data format and the insertion method is writing data row by row, submit the message consumption offset, end the insertion process and record the result.

Images