A method and system for intelligent management of coal mine data

Abstract

An embodiment of the present invention discloses a method and system for intelligent management of coal mine data. Among them, the method is applied to a coal mine data storage and computing integrated center, and includes: obtaining at least one of the following original coal mine data: coal mine data transmitted in real time by underground control transfer equipment, coal mine data stored in mobile storage equipment, and coal mine data transmitted by a coal mine geological hazard hidden danger transparency business system; processing the obtained original coal mine data through multiple processes, managing the original coal mine data and processing the coal mine data; and sending the managed coal mine data to the coal mine geological hazard hidden danger transparency business system. The technical solution provided by the embodiment of the present invention can be applied to scenarios where coal mine intelligence, transparent geological guarantee systems, and new quality productivity of coal mine data have high requirements for coal mine data storage and computing, and can realize unified management and efficient application of coal mine data.

Description

Technical Field

[0001] The present invention relates to the technical field of geophysical exploration, and particularly to a method and system for intelligent management of coal mine data. Background Art

[0002] In China's energy structure, the coal industry plays an indispensable role. With the improvement of society's requirements for environmental protection, resource utilization efficiency, and safe production with fewer or no people in coal mines, the development of coal mine intelligent technology has become particularly urgent and important. Currently, there are many problems in the transparent detection of hidden dangers of coal mine geological disasters, such as unclear multi-source and multi-field response mechanisms of geophysical fields of coal mine geological disaster sources, low detection accuracy, small detection range, slow geological information feedback response, low intelligent level of data processing, difficulty in realizing dynamic detection, untimely early warning response, insufficient geological information in mines, low geological modeling accuracy, and difficulty in updating static model attributes.

[0003] To address these challenges and promote the intelligent transformation of the coal mine industry, a scientific coal mine data acquisition, storage, processing, analysis, and interpretation solution is urgently needed to achieve mine geological transparency, and then realize green and intelligent precision coal mining. Summary of the Invention

[0004] In view of this, embodiments of the present invention provide a method and system for intelligent management of coal mine data to achieve efficient acquisition, storage, and real-time analysis of coal mine geological information, improve the transparency of coal mine geological information, and enhance the monitoring and early warning capabilities for geological disasters.

[0005] In a first aspect, embodiments of the present invention provide a method for intelligent management of coal mine data, which is applied to a coal mine data storage and computing integrated center. The method includes:

[0006] Obtain at least one of the following original coal mine data: coal mine data transmitted in real time by underground control transfer equipment, coal mine data stored in mobile storage devices, and coal mine data transmitted by a coal mine geological disaster hidden danger transparency business system;

[0007] Process the obtained original coal mine data through multiple processes, manage the original coal mine data and process the coal mine data;

[0008] Send the managed coal mine data to a coal mine geological disaster hidden danger transparency business system.

[0009] Further, managing the original coal mine data and processing the coal mine data includes:

[0010] Through the data lake in the intelligent lake warehouse, the original coal mine data is stored in partitions according to the hot and cold levels: the original coal mine data belonging to the hot data category is stored in the distributed file system server and cache server composed of solid-state drives; the original coal mine data belonging to the cold data category is compressed and stored in the distributed file system server composed of hard disk drives;

[0011] Through the data warehouse in the intelligent lake warehouse, manage and process the coal mine data.

[0012] Furthermore, the classification of cold data and hot data is determined according to the following formula:

[0013]

[0014] wherein, The calculation formula of is as follows:

[0015]

[0016] is the access frequency of data D at the current moment t, W is the size of the set time sliding window, is the number of accesses to data D at moment i;

[0017] The calculation formula of is as follows:

[0018]

[0019] is the hot data threshold at the current moment t, is the hot data threshold at the previous moment t - 1, is the set smoothing coefficient, is the set importance weight of data D, and β is the set weight smoothing coefficient.

[0020] Furthermore, select a compression strategy from multiple compression strategies according to the following formula to compress the original coal mine data belonging to the cold data category:

[0021]

[0022] wherein, is the selected compression strategy, is the calculation cost of compression strategy is the compression efficiency of compression strategy is compression strategy is the execution time of compression strategy is compression strategy is the of compression strategy is the compression strategy is the real-time monitoring value of the current system resource usage during execution, is the weight coefficient.

[0023] Furthermore, through the data warehouse in the intelligent lake warehouse, manage and process coal mine data, including:

[0024] In the data storage layer of the data warehouse, obtain and store the operation data generated by preprocessing the original coal mine data;

[0025] In the detail layer of the data warehouse, obtain and store the data model that meets the business requirements of coal mine geological hazard transparency based on the operation data;

[0026] In the service layer of the data warehouse, receive and respond to data access requests initiated by external access interfaces based on the data stored in the data warehouse;

[0027] In the analytical data storage layer of the data warehouse, obtain and store the analytical data generated by multi-process processing of the original coal mine data.

[0028] Furthermore, multi-process process the obtained original coal mine data, including:

[0029] Analyze the obtained original coal mine data to obtain at least one of the following acceleration data for generating the storage data of the data warehouse in the intelligent lake warehouse: coal mine data category, association relationship between coal mine data, sharing mechanism of coal mine data, access rights of coal mine data;

[0030] Among them, the association between coal mine data is dynamically maintained in real time based on the association relationship dynamic update algorithm. In this algorithm:

[0031] The association strength update formula between coal mine data is as follows:

[0032]

[0033] is the data and The association strength at the current moment t; is the data and The association strength at the next moment t + 1; is the data and The similarity function; is the preset balance coefficient; is the data and The change rate at the current moment t; is the influence coefficient of the preset change rate;

[0034] The correlation threshold update formula between coal mine data is as follows:

[0035]

[0036] is the threshold for correlation judgment at the current moment when making the correlation judgment, is the threshold for correlation judgment at the previous moment when making the correlation judgment, is the preset update coefficient;

[0037] When exceeds there is a correlation between the coal mine data and at this time.

[0038] Furthermore, the original coal mine data obtained through multi-process processing includes:

[0039] The management node of the computing center schedules each computing node in the computing pool to execute computing tasks in a load-balanced manner to process the obtained original coal mine data and / or intermediate processed coal mine data that meet the set high computing power requirements;

[0040] Among them, the allocation function of the computing task is as follows:

[0041]

[0042] is the allocation time of the j-th computing task on the i-th computing node, is the computing weight of the j-th computing task, is the computing power of the i-th computing node, is the load prediction value of the i-th computing node at the next moment t + 1;

[0043] The load balancing adjustment formula is as follows:

[0044]

[0045] is the load ratio of the i-th computing node at the current moment t; is the computing power or load capacity of the i-th computing node at the current moment t; is the computing power or load capacity of the i-th computing node at the next moment t + 1 after load balancing adjustment; is the maximum value among the load ratios of all computing nodes at the current moment t.

[0046] Further, after sending the managed coal mine data to the coal mine geological hazard transparency business system, the method further includes:

[0047] After receiving the abnormal information feedback sent by the coal mine geological hazard transparency business system, notify the underground terminal.

[0048] In a second aspect, an embodiment of the present invention provides a coal mine data intelligent management system, which is applied to the coal mine data intelligent management method described in the first aspect above. The coal mine data intelligent management system includes a coal mine data storage and computing integrated center. Among them, the coal mine data storage and computing integrated center includes: a data acquisition module, which is used to acquire at least one of the following original coal mine data: coal mine data transmitted in real time by underground control transfer equipment, coal mine data stored in mobile storage equipment, and coal mine data transmitted by the coal mine geological hazard transparency business system; a data management module, which is used to process the acquired original coal mine data in multiple processes, manage the original coal mine data and process the coal mine data; a data usage module, which is used to send the managed coal mine data to the coal mine geological hazard transparency business system.

[0049] Further, the coal mine data intelligent management system further includes coal mine multi-field and multi-source sensors, underground control transfer equipment, and mobile storage equipment, where:

[0050] The first type of coal mine multi-field and multi-source sensor is used to collect the first original coal mine data and transmit the first original coal mine data to the underground control transfer equipment in real time;

[0051] The second type of coal mine multi-field and multi-source sensor is used to collect the second original coal mine data and periodically store the second original coal mine data in the mobile storage equipment. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the following drawings are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0053] Figure 1 It is a flowchart of a coal mine data intelligent management method provided by Embodiment 1 of the present invention;

[0054] Figure 2 It is a working schematic diagram of an intelligent lake warehouse cold and hot data optimization LRU algorithm provided by Embodiment 1 of the present invention;

[0055] Figure 3 It is a functional structure diagram of a coal mine data storage and computing integrated center provided by Embodiment 2 of the present invention;

[0056] Figure 4 Schematic diagram of the hierarchical structure of an intelligent lake warehouse provided in the second embodiment of the present invention;

[0057] Figure 5 Schematic diagram of the composition architecture of a computing center provided in the second embodiment of the present invention;

[0058] Figure 6 Schematic diagram of the structure of a coal mine data intelligent management system provided in the third embodiment of the present invention. Specific implementation manners

[0059] The embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0060] It should be clear that the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative work belong to the scope of protection of the present invention. The technical solutions provided by the present invention will be described in detail through the following embodiments.

[0061] Embodiment 1

[0062] This embodiment provides a method for intelligent management of coal mine data, which can be executed by a coal mine data storage and computing integrated center. Refer to Figure 1 , this method specifically includes the following steps 101-103.

[0063] Step 101, obtain at least one of the following original coal mine data: coal mine data transmitted in real time by underground control transfer equipment, coal mine data stored in mobile storage devices, and coal mine data transmitted by the coal mine geological disaster hidden danger transparency business system.

[0064] In this step, the original coal mine data refers to the basic, unprocessed or unanalyzed initial data collected in coal mine mining and related activities. These data are usually directly obtained through various underground monitors (such as monitoring equipment, sensors), investigations or observations. The types of original coal mine data can include geological exploration data, environmental monitoring data, production operation data, safety monitoring data, resource utilization data, and other auxiliary data. The original coal mine data can be obtained from the following multiple sources to provide basic data support for subsequent intelligent management and analysis:

[0065] a. Coal mine data transmitted in real time by underground control transfer equipment. This data source can be collected in real time by underground monitors with high-reliability communication capabilities and sent to the underground control transfer equipment;

[0066] b. Coal mine data stored in mobile storage devices, which is usually exported from underground monitors without reliable communication capabilities and stored in mobile storage devices (such as disks, USB flash drives, hard disks);

[0067] c. The coal mine data transmitted by the transparent business system for potential coal mine geological disasters. Such data can be sourced from the data collected by different business subsystems.

[0068] Step 102: Process the obtained raw coal mine data through multiple processes, manage the raw coal mine data and process the coal mine data.

[0069] In this step, a series of continuous and different types of processing operations are performed on the obtained raw coal mine data to achieve the set data analysis and mining goals. Among them, the processing operations can involve multiple aspects such as data access, data cleaning, data conversion, data integration, data analysis and modeling, data visualization, feedback and iteration. The data obtained from the multi-process processing can be called processed coal mine data, which is divided into intermediate processed coal mine data and final coal mine data. These data are obtained through a series of processing operations based on the raw coal mine data. The data generated by each processing operation is the input of the subsequent processing operation until finally generating support data that can be used for analysis or decision-making. The management of coal mine data can include the storage, backup, update, access control, sharing control, etc. of coal mine data. Preferably, a hybrid storage mode of local and cloud services is adopted to store the raw coal mine data and the processed coal mine data.

[0070] The embodiment of the present invention can adopt a "data + computing power" mode to enable division of labor and cooperation between the multi-process processing and management of coal mine data to ensure efficient utilization of data resources. The coal mine data storage and computing integrated center includes two major parts: the intelligent lake warehouse and the computing center. The intelligent lake warehouse is responsible for managing two types of data: raw coal mine data and processed coal mine data, ensuring the quality, security, and availability of the data. In terms of processing task allocation, the intelligent lake warehouse can execute some common and simple processing tasks, such as data integration and preliminary statistical analysis. For more complex data processing tasks, they are undertaken by the computing center. This division of labor can optimize resource utilization and ensure that complex computing tasks do not affect the data storage and basic processing efficiency. Correspondingly, the raw coal mine data obtained through multi-process processing includes:

[0071] Through the computing center, process the raw coal mine data and / or intermediate processed coal mine data that meet the set high computing power requirements.

[0072] Specifically, during implementation, through the management node of the computing center, schedule each computing node in the computing pool to execute computing tasks in a load-balanced manner to process the obtained raw coal mine data and / or intermediate processed coal mine data that meet the set high computing power requirements.

[0073] Exemplarily, managing the raw coal mine data and the processed coal mine data includes: managing the raw coal mine data through the data lake in the intelligent lake warehouse; managing the processed coal mine data through the data warehouse in the intelligent lake warehouse. The following elaborates in detail.

[0074] (1) Manage the original coal mine data through the data lake

[0075] The data lake is a centralized storage architecture that can receive and store various formats of original coal mine data, including structured data, semi-structured data, and unstructured data. Thus, a large amount of data can be efficiently managed and stored.

[0076] When managing the original coal mine data, the data lake can store the original coal mine data by partitioning according to the hot and cold levels. Specifically, an optimized LRU (Least Recently Use) algorithm can be used to distinguish between cold data and hot data in the original coal mine data. Among them, hot data refers to data that is frequently read and processed; cold data refers to data with a low access frequency and low usage rate. As Figure 2 shown, when a certain original coal mine data first enters the lake, it will be placed at the tail of the lowest-priority data queue 1 in the hot data storage area. Subsequently, when this original coal mine data is accessed, according to the LRU algorithm rule, this data will be moved from the tail to the head direction of data queue 1. When the priority of this original coal mine data is further improved according to the access times, this data will be moved to the tail of a higher-priority data queue (the example in the figure is data queue 2 or data queue 3) in the hot data storage area, and this data in the original data queue will be deleted. Similarly, when the priority of this original coal mine data is decreased according to the access times, this data will be moved to the head of the low-priority data queue in the hot data storage area, and this data in the original data queue will be deleted. The data elimination in the hot data storage area always starts from the tail of the lowest-priority data queue, and the eliminated tail data will be added to the head of the historical data queue. If the data is accessed in the historical data queue, the priority of this data will be recalculated and it will be added to the corresponding-priority data queue in the hot data storage area, otherwise it will be completely eliminated from the hot data storage area according to the LRU algorithm and transferred to the cold data storage area.

[0077] Based on the above LRU algorithm, an improved adaptive hot and cold data partitioning algorithm is introduced to dynamically adjust the partitioning criteria for hot and cold data. The adaptive hot and cold data partitioning algorithm defines the sliding window function of the access frequency as follows:

[0078]

[0079] Among them, is the access frequency of data D at the current time t, W is the set time sliding window size, is the access times of data D at the current time i.

[0080] Adaptive adjustment threshold:

[0081]

[0082] Among them, is the thermal data threshold at the current moment t, is the thermal data threshold at the previous moment t-1, is the set smoothing coefficient, which is used to balance the influence of historical thermal data threshold and current data, is the importance weight of the set data D, and β is the set weight smoothing coefficient.

[0083] Cold data and hot data are classified according to the following formula:

[0084]

[0085] Preferably, the original coal mine data belonging to the hot data category is stored in the distributed file system HDFS server and cache server composed of solid state drives SSD; the original coal mine data belonging to the cold data category is compressed and stored in the distributed file system HDFS server composed of hard disk drives HDD to save the storage space and cost of a large amount of coal mine data. Exemplarily, the distributed file system server composed of solid state drives can store all the hot data and ensure that the most frequently accessed data is always in the cache server. When external data is accessed, the storage in the cache server is preferentially read. Regarding data compression, the Deflate compression algorithm can be used to compress seismic exploration.segy format data, the LZ78 algorithm can be used to compress unformatted or processed binary data, such as geological radar detection data.raw format data; the Huffman coding can be used to compress the ASCII text format.las format data of well logging data, etc.

[0086] Among them, for the compression of the original coal mine data of the above cold data category, its compression strategy can be dynamically adjusted according to various factors such as storage space and processing requirements. Exemplarily, a compression strategy can be selected from multiple compression strategies for use:

[0087]

[0088] Among them, is the selected compression strategy, is the computational cost of the compression strategy is the compression efficiency (compression ratio) of the compression strategy is the execution time of the compression strategy is the compression strategy The real-time monitoring values of the current system resource usage (CPU, memory, GPU, etc.) during execution, is the set weight coefficient.

[0089] Preferably, the weight coefficient can also be dynamically optimized over time according to the following formula:

[0090]

[0091] where L is the optimization objective function, which is preset according to actual application requirements (such as space minimization or time minimization); is the current time used weight coefficient, is the next time used weight coefficient.

[0092] The above hot and cold storage mechanism has the following advantages:

[0093] Improve access efficiency: Hot data is stored in a distributed file system server and a cache server composed of solid-state drives. These storage media have faster read and write speeds, which can significantly improve the access efficiency of hot data;

[0094] Reduce costs: Cold data is stored in a distributed file system server composed of hard disk drives with lower costs. The storage space requirement is reduced through compressed storage, thereby reducing the overall storage cost;

[0095] Optimize resource utilization: By storing data with different access frequencies on different types of storage media, effective allocation of resources is achieved, ensuring fast access to hot data while also reducing the cost of cold data storage;

[0096] Simplify data management: By adopting an optimized LRU algorithm to automatically distinguish between hot and cold data, the data management process is simplified, reducing the need for manual intervention;

[0097] Enhance data availability: Hot data is located in fast-access storage, ensuring high availability and low-latency access. Although cold data has a lower access frequency, it can still be quickly retrieved through the distributed file system;

[0098] Improve data security: Through partitioned storage, different security measures can be taken for different types of data, improving the overall data security.

[0099] (2) Manage and process coal mine data through a data warehouse

[0100] ①In the operation data storage layer of the data warehouse, obtain and store the operation data generated by preprocessing the original coal mine data.

[0101] Among them, the preprocessing includes simple cleaning, deduplication, format conversion, etc.

[0102] ②In the detail layer of the data warehouse, obtain and store the data model that meets the business requirements of the transparency of coal mine geological disaster hazards based on the operation data.

[0103] Among them, the computing center can be called to use existing big data and artificial intelligence technologies to clean, process, integrate, aggregate, etc. the operation data in the operation data storage layer, and build a data model that meets the business requirements of the transparency of coal mine geological disaster hazards.

[0104] ③In the service layer of the data warehouse, receive and respond to data access requests initiated by external access interfaces based on the data stored in the data warehouse.

[0105] Among them, the service layer defines access interfaces for the data stored in the data warehouse and provides various types of data services externally, such as data query, statistics, report generation, analysis, etc. Optionally, the intermediate processed coal mine data can also be further processed through ETL operations, etc. to provide real-time data services for upper-layer applications.

[0106] ④In the analytical data storage layer of the data warehouse, obtain and store the analytical data generated by multi-process processing of the original coal mine data.

[0107] The analytical data is a kind of structured and standardized data generated after multiple processing processes such as cleaning, integration, transformation, and modeling of the original coal mine data, which can reveal potential patterns, trends, and relationships in the data. Building and storing specially optimized analytical data can achieve efficient and convenient query of multi-source coal mine data, and at the same time have the ability of quick response, providing data support for the transparency business of coal mine geological disaster hazards.

[0108] As a preferred implementation method, the original coal mine data obtained by multi-process processing includes:

[0109] Analyze the obtained original coal mine data to obtain at least one of the following acceleration data for generating the stored data in the data warehouse of the intelligent lake warehouse: coal mine data category, association relationship between coal mine data, sharing mechanism of coal mine data, access permission of coal mine data.

[0110] Among them, the association relationship between coal mine data can be the data lineage relationship, which is specifically obtained by parsing the original coal mine data through Apache Flink, Apache Spark, and Trino. The data lineage relationship of coal mine data covers the entire process from coal mine data acquisition to multi-process processing. In addition, the association relationship can also be a coal mine data knowledge graph established using the graph database Neo4J.

[0111] During the analysis process of coal mine data, the association relationship between data changes over time and space. A dynamic update algorithm for association relationships based on a graph database can be introduced to maintain the relevance between data in real time. Exemplarily, in the dynamic update algorithm for association relationships, the association strength and association threshold between coal mine data change over time, specifically as follows:

[0112] The update formula for the association strength between coal mine data is as follows:

[0113]

[0114] Where: is the association strength of data and at the current time t; is the association strength of data and at the next time t + 1; is the similarity function of data and ; is the balance coefficient between historical data and the current calculation result, which is preset; is the change rate of data and at the current time t; is the influence coefficient of the change rate, which is preset;

[0115] The update formula for the association threshold is as follows:

[0116]

[0117] Where, is the threshold for association judgment at the current time t, is the threshold for association judgment at the previous time t, is the preset update coefficient.

[0118] Only when exceeds will its connection be maintained in the graph database, that is, there is an association between coal mine data and .

[0119] Step 103: Send the managed coal mine data to the coal mine geological disaster hidden danger transparency business system.

[0120] In this step, the managed coal mine data can be actively broadcast to the coal mine geological disaster hidden danger transparency business system regularly. The specific broadcast object can be the intelligent platform in the system, which is used to display and update the coal mine geological disaster hidden danger transparency map. Of course, those skilled in the art should understand that the managed coal mine data corresponding to the request can also be sent to the subsystem after receiving the data access request of each subsystem in the coal mine geological disaster hidden danger transparency business system. The embodiments of the present invention do not make specific limitations on this.

[0121] Based on the above solution, as an optional implementation manner, after step 103, it further includes: after receiving the abnormal information feedback sent by the coal mine geological disaster hidden danger transparency business system, notify the underground terminal. Among them, the underground terminal refers to the terminal equipment used by the equipment or personnel operating in the coal mine underground, which can include explosion-proof mobile phones, explosion-proof tablets, explosion-proof laptops, etc. By immediately notifying the underground terminal, it can ensure that miners have sufficient awareness of potential geological disasters and take corresponding preventive measures, thereby reducing the risk of accidents.

[0122] Embodiment 2

[0123] This embodiment provides a coal mine data storage and computing integrated center, which can be used to execute the coal mine data intelligent management method described in the embodiments of the present invention. See Figure 3 , the coal mine data storage and computing integrated center specifically includes the following modules:

[0124] Data acquisition module 301, which is used to acquire at least one of the following original coal mine data: the coal mine data transmitted in real time by the underground control transfer equipment, the coal mine data stored in the mobile storage device, and the coal mine data transmitted by the coal mine geological disaster hidden danger transparency business system;

[0125] Data management module 302, which is used to process the acquired original coal mine data in multiple processes, manage the original coal mine data and process the coal mine data;

[0126] Data usage module 303, which is used to send the managed coal mine data to the coal mine geological disaster hidden danger transparency business system.

[0127] Furthermore, the data management module 302 includes: a first management sub-module, which is used to manage the original coal mine data through the data lake in the intelligent lake warehouse; a second management sub-module, which is used to manage the processed coal mine data through the data warehouse in the intelligent lake warehouse.

[0128] Further, the first management sub-module is used to manage the original coal mine data through the data lake in the intelligent lake warehouse, specifically including: storing the original coal mine data in partitions according to the hot and cold levels through the data lake in the intelligent lake warehouse.

[0129] Further, the first management sub-module is used to store the original coal mine data in partitions according to the hot and cold levels through the data lake in the intelligent lake warehouse, specifically including:

[0130] Storing the original coal mine data belonging to the hot data category in the distributed file system server and cache server composed of solid-state drives;

[0131] Compressively storing the original coal mine data belonging to the cold data category in the distributed file system server composed of hard disk drives.

[0132] Further, the second management sub-module is used to manage the processed coal mine data through the data warehouse in the intelligent lake warehouse, specifically including:

[0133] Operating on the data storage layer of the data warehouse to obtain and store the operation data generated by preprocessing the original coal mine data;

[0134] Obtaining and storing, in the data warehouse detail layer, the data model that meets the business requirements of coal mine geological disaster hidden danger transparency based on the operation data;

[0135] Receiving and responding to data access requests initiated externally through the access interface for the data stored in the data warehouse at the data warehouse service layer;

[0136] Obtaining and storing, in the data warehouse analytical data storage layer, the analytical data generated by multi-process processing of the original coal mine data.

[0137] Further, the data management module 302 further includes:

[0138] A data acceleration sub-module, which is used to analyze the obtained original coal mine data to obtain at least one of the following acceleration data for generating the stored data in the data warehouse in the intelligent lake warehouse: coal mine data category, association relationship between coal mine data, sharing mechanism of coal mine data, access permission of coal mine data.

[0139] Further, the data management module 302 is used to manage the original coal mine data and the processed coal mine data, specifically including: adopting a hybrid storage mode of local and cloud services to store the original coal mine data and the processed coal mine data.

[0140] Further, the data management module 302 is used to multi-process the obtained original coal mine data, specifically including: processing the original coal mine data and / or intermediate processed coal mine data that meet the set high computing power requirements through the computing center.

[0141] Further, the data management module 302 is used to process the acquired raw coal mine data and / or intermediate processed coal mine data that meet the set high computing power requirements through the computing center, specifically including:

[0142] Through the management node of the computing center, schedule each computing node in the computing pool to execute computing tasks in a load-balanced manner to process the acquired raw coal mine data and / or intermediate processed coal mine data that meet the set high computing power requirements.

[0143] In order to process coal mine data more efficiently, the computing tasks can be dynamically allocated based on a preset parallel processing optimization algorithm to improve the processing efficiency. In this algorithm, the allocation function of the computing tasks is as follows:

[0144]

[0145] Wherein, is the allocation time of the jth computing task on the ith computing node, is the computing weight of the jth computing task, is the computing power of the ith computing node, is the load prediction value of the ith computing node at the next moment t + 1.

[0146] The load balancing adjustment formula is as follows:

[0147]

[0148] Wherein, is the load ratio of the ith computing node at the current moment t (i.e., the current load of this node), representing the actual load level of the ith computing node at the current moment t; dynamically adjust the load distribution of the computing nodes to achieve load balancing. is the computing power or load capacity of the ith computing node at the current moment t; is the computing power or load capacity of the ith computing node at the next moment t + 1 after load balancing adjustment. The maximum value among the load ratios of all computing nodes k at the current moment t provides a standard for comparing with the load ratio of the ith computing node to help determine the adjustment amplitude.

[0149] The above load balancing adjustment formula is used to update the computing power according to the load ratio of the current computing node relative to the maximum load of other computing nodes. Adjust the computing power of the computing node by increasing the ratio of the current computing node's load ratio to the maximum load ratio. If the load of the current computing node is close to the maximum load, its computing power will increase more to help balance the load.

[0150] Further, the coal mine data storage and computing integrated center further includes a notification module 304, which is used to: after receiving the abnormal information feedback sent by the coal mine geological disaster hidden danger transparency business system, notify the underground terminal.

[0151] The above is an elaboration of the coal mine data storage and computing integrated center from the perspective of the implementation of functional modules. To understand the structure and functions of the coal mine data storage and computing integrated center more clearly, the coal mine data storage and computing integrated center will be further explained in detail from different levels in the data flow and processing process. The coal mine data storage and computing integrated center includes an intelligent lake warehouse and a computing center. Among them, referring to Figure 4 , the intelligent lake warehouse includes the following layers:

[0152] (1) Coal mine multi-source data layer

[0153] It is composed of coal mine data in multiple subsystem databases (including MySQL, PostgreSQL, HBase, Redis, MongoDB, InfluxDB, Neo4j, PostGIS, Kdb, Elasticsearch, etc.) in the existing coal mine geological disaster hidden danger transparency business system, coal mine data stored in mobile storage devices (USB flash drives, disks, hard disks, etc.), and real-time monitored coal mine data streams;

[0154] (2) Data acquisition layer

[0155] Data is accessed from the coal mine multi-source data layer through Apache Flink, Apache Spark, and Trino; on the one hand, it accesses the real-time monitored data stream input in the coal mine underground and the data stored in mobile storage devices, and on the other hand, it accesses the data connection libraries of existing subsystems, ensuring the daily stable operation of existing systems and the data transition and upgrade;

[0156] (3) Data management and acceleration layer

[0157] Analyze the original coal mine data accessed by the data acquisition layer to obtain at least one of the following acceleration data for generating the storage data of the data warehouse in the intelligent lake warehouse: coal mine data category (i.e., data content), the association relationship between coal mine data (the example in the figure is data lineage), the sharing mechanism of coal mine data, and the access rights of coal mine data;

[0158] (4) Lake warehouse storage layer

[0159] Establish a hybrid storage cluster of the distributed file system HDFS and the object storage AWS S3 to achieve a hybrid storage mode of local and cloud services, and store the original coal mine data and acceleration data;

[0160] (5) Resource management layer

[0161] The resource scheduling and management of the intelligent coal mine lake storage can be achieved through the resource managers Yarn and Kubernetes;

[0162] (6) Data lake

[0163] For the specific description of the data lake, please refer to Embodiment 1, which will not be elaborated here;

[0164] (7) Data warehouse

[0165] Read the original coal mine data and acceleration data from the lake storage layer and connect them to the data warehouse. The data warehouse includes the following four layers of operations: Operational Data Store (ODS), Data Warehouse Detail (DWD), Data Warehouse Summary (DWS), and Analytical Data Store (ADS); for the specific description, please refer to the relevant description in Embodiment 1, which will not be elaborated here;

[0166] (8) Application layer

[0167] One is to provide an interface for accessing the original coal mine data in the data lake, which can serve upper-layer applications for statistical analysis (such as gas monitoring), data mining, and data fusion, etc.; the other is to provide an interface for accessing the analytical data in the data warehouse, which can serve applications such as transparent geological modeling of geological hazard hidden dangers, dynamic interpretation of all-element multi-physical fields of geological hazard hidden dangers, aggregation and mining of multi-field multi-attribute data, and sharing of coal mine exploration and interpretation data.

[0168] Computing center, such as Figure 5 As shown, it may include Data Center A and Data Center B, and the two data centers play a disaster tolerance role; each data center consists of multiple computing pools (2 in the figure as an example), and each computing pool consists of a management node and multiple computing nodes (3 in the figure as an example). The management node is responsible for task allocation to ensure the load balance of the computing nodes in the data center, and the computing nodes are responsible for processing the original coal mine data and / or intermediate processed coal mine data that meet the set high computing power requirements, such as transparent seismic exploration inversion interpretation and attribute analysis of geological hazard hidden dangers (calculating seismic attributes, such as instantaneous amplitude, phase, frequency, etc.), and processing and interpretation applications of machine learning and artificial intelligence for seismic exploration data.

[0169] The coal mine data storage and computing integrated center provided in this embodiment belongs to the same inventive concept as the foregoing method embodiment. For the technical details not described in this embodiment, please refer to the relevant description in the foregoing method embodiment, which will not be elaborated here.

[0170] Embodiment 3

[0171] This embodiment provides a coal mine data intelligent management system, which is applied to the coal mine data intelligent management method in the embodiments of the present invention. Refer to Figure 6 , the coal mine data intelligent management system includes a coal mine data storage and computing integrated center: an intelligent lake warehouse and a computing center.

[0172] Furthermore, the coal mine data intelligent management system may further include coal mine multi-field and multi-source sensors, underground control transfer equipment, and mobile storage equipment (not shown in the figure), where:

[0173] The first type of coal mine multi-field and multi-source sensor is used to collect the first original coal mine data and transmit the first original coal mine data to the underground control transfer equipment in real time;

[0174] The second type of coal mine multi-field and multi-source sensor is used to collect the second original coal mine data and periodically store the second original coal mine data in the mobile storage equipment.

[0175] Among them, the coal mine multi-field and multi-source sensor, as a kind of underground monitor, can obtain various types of coal mine data from the following aspects: gravity, magnetic method, electrical method, seismic, geology, surveying and mapping, etc. Specifically, means such as microgravity measurement, transient electromagnetic method exploration, high-density electrical method exploration, ground-penetrating radar detection, lidar SLAM scanning, three-dimensional / four-dimensional seismic exploration, microseismic monitoring, UWB inertial navigation positioning, etc. can be adopted. The first type of coal mine multi-field and multi-source sensor has high-reliability communication ability and can transmit data to the underground control transfer equipment in real time through WIFI. The second type of coal mine multi-field and multi-source sensor does not have high-reliability communication ability.

[0176] The underground control transfer equipment receives the original coal mine data transmitted by the coal mine multi-field and multi-source sensors and performs real-time processing and interpretation, and stores the original coal mine data and the real-time interpretation data results in a buffer (the buffer mainly avoids the data transmission pressure of the underground control transfer equipment caused by real-time massive data). The buffer is transmitted to the coal mine data storage and computing integrated center in real time through optical fiber. And, the underground control transfer equipment can also control the operation of mining equipment (such as a shearer).

[0177] In addition, the coal mine data storage and computing integrated center can be connected to the intelligent platform of the coal mine geological disaster hidden danger transparency business system for: regularly and actively broadcasting the managed coal mine data to the intelligent platform for the intelligent platform to display and update the coal mine geological disaster hidden danger transparency map; after receiving the abnormal information feedback sent by the intelligent platform, notifying the underground terminal.

[0178] The coal mine data storage and computing integrated center can also be connected to the subsystems of the coal mine geological disaster hidden danger transparency business system for: receiving the coal mine data transmitted by the subsystems of the business system, performing multi-process processing and management on the coal mine data; and sending the managed coal mine data to the subsystems of the business system. Among them, the business subsystems can include one or more of the tunneling system, the hierarchical control system, the comprehensive monitoring system, etc.

[0179] In summary, the technical solution provided by the embodiments of the present invention can realize the scientific storage management, real-time dynamic processing interpretation and practical application of multi-source heterogeneous data of coal mine geological disaster hidden dangers.

[0180] It should be noted that in this article, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the term "including", "comprising" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements, but also includes other elements not expressly listed, or also includes elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "including a..." does not exclude the existence of another identical element in the process, method, article or device including the element.

[0181] In the embodiments of the present invention, the term "and / or" describes the association relationship of associated objects, indicating that three relationships can exist. For example, A and / or B can represent: A exists alone, A and B exist simultaneously, and B exists alone. The character " / " generally represents an "or" relationship between the associated objects before and after.

[0182] Each embodiment in this specification is described in a related manner. The same or similar parts between the embodiments can be referred to each other, and the differences between each embodiment and other embodiments are emphasized.

[0183] In particular, for the device embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0184] For the convenience of description, the above device is described by dividing it into various units / modules according to functions. Of course, when implementing the present invention, the functions of the various units / modules can be realized in the same or multiple software and / or hardware.

[0185] Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by instructing relevant hardware through a computer program. The program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above various methods. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM), etc.

[0186] As described above, the above are only specific embodiments of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by those skilled in the art within the technical scope disclosed by the present invention should be covered within the protection scope of the present invention. Therefore, the protection scope of the present invention should be subject to the protection scope of the claims.

Claims

1. A method for intelligent management of coal mine data, characterized in that: Applied to a coal mine data storage and computing integrated center, the method includes: Obtain at least one of the following original coal mine data: coal mine data transmitted in real time by underground control transfer equipment, coal mine data stored in mobile storage devices, and coal mine data transmitted by a coal mine geological disaster hidden danger transparency business system; Multi-process processing of the acquired raw coal mine data, management of raw coal mine data and processing of coal mine data; Send the managed coal mine data to the coal mine geological disaster hidden danger transparency business system; The managing of raw coal mine data and processing of coal mine data include: managing and processing coal mine data through a data warehouse in an intelligent lake warehouse; The data warehouse in the intelligent lake warehouse is used to manage and process coal mine data, including: In the data warehouse operation data storage layer, the operation data generated by preprocessing the original coal mine data is obtained and stored; At the data warehouse detail layer, a data model generated based on the operation data and meeting the business requirements for transparency of geological hazards in coal mines is obtained and stored; At the data warehouse service layer, receive and respond to data access requests initiated by external access interfaces based on data stored in the data warehouse; In the data warehouse analytical data storage layer, analytical data generated by multi-process processing of original coal mine data is obtained and stored; The data model obtained and stored in the data warehouse detail layer includes: cleaning, processing, integrating and aggregating the operation data in the operation data storage layer to construct a data model that meets the business needs of transparent geological disaster hazards in coal mines.

2. The method according to claim 1, characterized in that Manage raw coal mine data and process coal mine data, including: The original coal mine data is stored in the data lake in the smart lake warehouse according to the hot and cold levels: the original coal mine data belonging to the hot data category is stored in the distributed file system server and cache server composed of solid-state hard drives; the original coal mine data belonging to the cold data category is compressed and stored in the distributed file system server composed of hard disk drives; Coal mine data is managed and processed through the data warehouse in the smart lake warehouse.

3. The method according to claim 2, characterized in that Cold Data and thermal data The classification is determined according to the following formula: ； in, The calculation formula is as follows: ； For the current moment t Time data D The frequency of visits, W is the set time sliding window size, For at the moment i Time data D Number of visits; The calculation formula is as follows: ； For the current moment t The hot data threshold at is the hot data threshold at the previous time t-1, is the set smoothing coefficient, For setting data D The importance weight of , β is the set weight smoothing coefficient.

4. The method according to claim 2, characterized in that: According to the following formula, from multiple compression strategies Select a compression strategy in , compress the original coal mine data belonging to the cold data category: ； in, is the selected compression strategy, Compression strategy The computational cost of Compression strategy The compression efficiency, Compression strategy The execution time, Compression strategy During execution, the current system resource usage is monitored in real time. is the weight coefficient.

5. The method according to claim 2, characterized in that: The raw coal mine data obtained by multi-process processing includes: Analyze the acquired original coal mine data to obtain at least one of the following acceleration data, which is used to generate storage data of the data warehouse in the intelligent lake warehouse: coal mine data category, association relationship between coal mine data, sharing mechanism of coal mine data, and access rights to coal mine data; Among them, the correlation between coal mine data is maintained in real time based on the dynamic update algorithm of correlation relationship. In this algorithm: The update formula of the correlation strength between coal mine data is as follows: ； For data and At the present moment t The strength of association; For data and At the next moment t+1 The strength of association; For data and Similarity function of ; is the preset balance coefficient; For data and At the present moment t The rate of change of is the influence coefficient of the preset change rate; The update formula of the correlation threshold between coal mine data is as follows: ； For the current moment The threshold value of correlation judgment is For the previous moment The threshold value of correlation judgment is is a preset update coefficient; when Exceed When the coal mine data and There is a correlation between them.

6. The method according to any one of claims 1 to 5, characterized in that The raw coal mine data obtained by multi-process processing includes: Through the management node of the computing center, the computing nodes in the computing pool are scheduled to perform computing tasks in a load-balanced manner to process the acquired raw coal mine data and / or intermediate processed coal mine data that meet the set high computing power requirements; Among them, the allocation function of the computing task is as follows: ； For the j The computational task is i The allocation time on the computing nodes, For the j The computational weight of a computational task, For the i The computing power of the computing nodes, For the i The computing nodes at the next moment t+1 The load forecast value; The load balancing adjustment formula is as follows: ； For the i The computing nodes at the current time Load ratio; For the i The computing nodes at the current time The computing power or load capacity; After load balancing adjustment, i The computing nodes at the next moment +1 for computing power or load capacity; For all computing nodes at the current time The maximum value of the load ratio.

7. The method according to claim 1, characterized in that After sending the managed coal mine data to the coal mine geological disaster hidden danger transparency business system, the method further includes: After receiving the abnormal information feedback sent by the coal mine geological disaster hidden danger transparency business system, notify the underground terminal.

8. An intelligent coal mine data management system, characterized in that: Applied to the coal mine data intelligent management method as described in claims 1-7, the coal mine data intelligent management system comprises a coal mine data storage and computing integrated center; Among them, the coal mine data storage and computing integrated center includes: a data acquisition module, used to obtain at least one of the following original coal mine data: coal mine data transmitted in real time by underground control transfer equipment, coal mine data stored in mobile storage devices, and coal mine data transmitted by the coal mine geological disaster hidden danger transparency business system; a data management module, used for multi-process processing of the acquired original coal mine data, management of the original coal mine data and processing of the coal mine data; a data usage module, used to send the managed coal mine data to the coal mine geological disaster hidden danger transparency business system.

9. The intelligent coal mine data management system according to claim 8, characterized in that: The coal mine data intelligent management system also includes coal mine multi-field multi-source sensors, underground control transfer equipment and mobile storage equipment, among which: The first type of coal mine multi-field multi-source sensor is used to collect the first original coal mine data and transmit the first original coal mine data to the underground control transfer equipment in real time; The second type of coal mine multi-field multi-source sensor is used to collect the second original coal mine data and regularly store the second original coal mine data in a mobile storage device.

Images