A coal mine geological big data processing system and method

Abstract

The embodiment of the present invention discloses a coal mine geological big data processing system and method, which relates to the field of coal mine geological digital technology, and can realize efficient processing of coal mine geological big data. The system includes: a receiving unit, which is used to receive coal mine geological big data processing tasks; a processing unit, which is used to allocate storage nodes and / or computing nodes according to preset conditions to complete the coal mine geological big data processing tasks; wherein the processing unit includes: a resource layer, which is used to manage each node of the storage layer and the computing layer; a storage layer, which is used to analyze the storage requirements of coal mine geological big data tasks and meet the storage and access requirements of coal mine geological big data; a computing layer, which is used to meet the computing and processing requirements of coal mine geological big data tasks; a scheduling layer, which is used for scheduling and management of coal mine geological big data processing tasks, to achieve efficient execution of tasks and optimal utilization of resources. The present invention can be used in the coal mining industry.

Description

Technical Field

[0001] The present invention relates to the field of coal mines, and in particular to a coal mine geological big data processing system and method. Background Art

[0002] With the development of exploration technology, coal mine geological data has shown explosive growth, and the data scale can reach PB and EB levels. Coal mine geological data has entered the era of big data. On the one hand, facing such a huge amount of data, simply expanding the storage capacity of the computer device file system by increasing the number of hard disks can no longer meet the current storage needs of massive multi-source heterogeneous data. On the other hand, the bottleneck of single computing nodes in data processing and analysis means that when processing large-scale data, the processing capacity of a single computing node is limited and it is impossible to efficiently complete complex data processing and analysis tasks. The current system is difficult to meet the needs of the times for massive, multi-source heterogeneous coal mine geological data storage management and computing analysis. A solution for quickly and efficiently deploying and scheduling coal mine geological big data systems is needed to improve the utilization value of data. Summary of the invention

[0003] In view of this, an embodiment of the present invention provides a coal mine geological big data processing system and method, which can realize efficient processing of coal mine geological big data.

[0004] In a first aspect, an embodiment of the present invention provides a coal mine geoscience big data processing system, comprising:

[0005] A receiving unit, used for receiving coal mine geoscience big data processing tasks;

[0006] A processing unit, used for allocating the coal mine geoscience big data processing task to storage nodes and / or computing nodes according to preset conditions to complete the coal mine geoscience big data processing task;

[0007] Wherein, the processing unit includes:

[0008] The resource layer is used to manage the nodes of the storage layer and computing layer;

[0009] The storage layer is used to analyze the storage requirements of coal mine geological big data and meet the storage and access requirements of coal mine geological big data;

[0010] The computing layer is used to meet the computing and processing requirements of coal mine geological big data;

[0011] The scheduling layer is used for scheduling and management of coal mine geological big data processing tasks to achieve efficient execution of tasks and optimal utilization of resources.

[0012] Optionally, the storage layer includes:

[0013] A classification module, used for dividing the collected coal mine data into a plurality of sub-data sets according to a first rule, and assigning a unique identifier to each sub-data set;

[0014] A mapping module, used to establish a mapping relationship between the sub-dataset and the original sub-cluster of the storage layer according to the second rule;

[0015] A processing module is used to complete distributed storage and parallel processing of the sub-datasets in the sub-clusters with established mapping relationships.

[0016] The dynamic adjustment module is used to dynamically adjust the data of the sub-datasets and the mapping relationship between the sub-datasets and the sub-clusters according to the real-time operating status of the coal mine geological big data processing system and user requests.

[0017] Optionally, the computing layer includes:

[0018] The self-detection module is used to automatically migrate and reschedule tasks through automatic detection and failover mechanisms when computing nodes in the computing layer fail or fail;

[0019] The dynamic adjustment module is used to monitor and dynamically adjust the resource utilization and load conditions of the computing nodes in the computing layer to achieve load balancing of the computing nodes and improve overall performance and resource utilization efficiency.

[0020] Optionally, the scheduling layer includes:

[0021] The load balancing module is used to monitor the load and resource utilization of each node in the storage layer and computing layer, and dynamically adjust and allocate computing and storage tasks to the optimal node through load balancing strategies;

[0022] The failover module is used to monitor computing nodes and storage nodes in real time, and adopts fault detection and failover mechanisms. When a node failure or failure is found, it automatically migrates and reschedules tasks.

[0023] In a second aspect, an embodiment of the present invention provides a method for processing coal mine geological big data, including:

[0024] Accept coal mine geological big data processing tasks;

[0025] Allocating the coal mine geoscience big data processing task to storage nodes and / or computing nodes according to preset conditions to complete the coal mine geoscience big data processing task;

[0026] The method of allocating the coal mine geoscience big data processing task to storage nodes and / or computing nodes according to preset conditions to complete the coal mine geoscience big data processing task includes:

[0027] Manage the storage nodes for coal mine geological big data and the computing nodes for computing and processing big data;

[0028] Analyze the storage requirements of coal mine geological big data, and store and access coal mine geological big data;

[0029] Calculate and process coal mine geological big data;

[0030] Schedule and manage coal mine geological big data processing tasks to achieve efficient task execution and optimal utilization of resources.

[0031] Optionally, analyzing the storage requirements of coal mine geological big data, storing and accessing the coal mine geological big data, includes:

[0032] Dividing the collected coal mine data into a plurality of sub-data sets according to a first rule, and assigning a unique identifier to each sub-data set;

[0033] Establishing a mapping relationship between the sub-dataset and the original sub-cluster of the storage layer according to the second rule;

[0034] The sub-datasets are distributedly stored and processed in parallel in the sub-clusters that establish mapping relationships.

[0035] According to the real-time operation status of the coal mine geological big data processing system and the user request situation, the data of the sub-datasets and the mapping relationship between the sub-datasets and the sub-clusters are dynamically adjusted.

[0036] Optionally, the calculation and processing of coal mine geological big data includes:

[0037] When a computing node fails or fails, tasks are automatically migrated and rescheduled through automatic detection and failover mechanisms;

[0038] Monitor and dynamically adjust the resource utilization and load conditions of computing nodes to achieve load balancing of computing nodes and improve overall performance and resource utilization efficiency.

[0039] Optionally, the scheduling and management of coal mine geological big data processing tasks to achieve efficient execution of tasks and optimal utilization of resources includes:

[0040] Monitor the load and resource utilization of computing nodes and storage nodes, and dynamically adjust and allocate computing and storage tasks to the optimal nodes through load balancing strategies;

[0041] The computing nodes and storage nodes are monitored in real time, and fault detection and failover mechanisms are adopted. When node faults or failures are detected, tasks are automatically migrated and rescheduled.

[0042] The coal mine geological big data processing system provided by the embodiment of the present invention includes: a receiving unit for receiving coal mine geological big data processing tasks; a processing unit for allocating the coal mine geological big data processing tasks to storage nodes and / or computing nodes according to preset conditions to complete the coal mine geological big data processing tasks; wherein the processing unit includes: a resource layer for managing each node of the storage layer and the computing layer; a storage layer for analyzing the storage requirements of coal mine geological big data to meet the storage and access requirements of coal mine geological big data; a computing layer for meeting the computing and processing requirements of coal mine geological big data; a scheduling layer for scheduling and managing coal mine geological big data processing tasks to achieve efficient execution of tasks and optimal use of resources. By allocating appropriate storage nodes and / or computing nodes to the coal mine geological big data processing tasks, efficient processing of coal mine geological big data is achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0044] Figure 1 A schematic diagram of the structure of a coal mine geoscience big data processing system provided by an embodiment of the present invention;

[0045] Figure 2 Provides an overall architecture diagram of a processing unit for an embodiment of the present invention;

[0046] Figure 3 A diagram of a resource layer architecture provided by an embodiment of the present invention;

[0047] Figure 4 A resource layer deployment flow chart provided for an embodiment of the present invention;

[0048] Figure 5 A method for processing coal mine geological big data is provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0049] A coal mine geological big data processing system provided by an embodiment of the present invention is described in detail below with reference to the accompanying drawings.

[0050] It should be clear that the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0051] Figure 1 A schematic diagram of a structure of a coal mine geoscience big data processing system provided by an embodiment of the present invention. Figure 1 As shown, the big data processing system of this embodiment may include:

[0052] A receiving unit 11, used for receiving a big data processing task;

[0053] A processing unit 12 is used to allocate the big data processing task to storage nodes and / or computing nodes according to preset conditions to complete the big data processing task;

[0054] Among them, the processing unit 12 includes: a resource layer, which is used to manage each node of the storage layer and the computing layer; a storage layer, which is used to analyze the storage requirements of coal mine geological big data and meet the storage and access requirements of coal mine geological big data; a computing layer, which is used to meet the computing and processing requirements of coal mine geological big data; a scheduling layer, which is used to schedule and manage coal mine geological big data processing tasks to achieve efficient execution of tasks and optimal utilization of resources.

[0055] The allocation of storage nodes and / or computing nodes for the coal mine geological big data processing task according to preset conditions can generally be in the following situations: 1. According to the needs of the coal mine geological big data processing task, such as the task needs to store data or access the data of the storage node, the task is allocated to the corresponding storage node; 2. According to the load of each node in the current system, such as the task needs to allocate computing nodes, there are three computing nodes in the current system, node one is occupied 90%, node two is occupied 95%, and node three is occupied 20%, then the task will be allocated to node three first; 3. Task splitting and parallel computing, such as splitting the task into multiple subtasks, and assigning them to different computing nodes for parallel execution, to realize the task of parallel computing between multiple nodes.

[0056] The coal mine geological big data processing system provided by the embodiment of the present invention includes: a receiving unit for receiving coal mine geological big data processing tasks; a processing unit for allocating the coal mine geological big data processing tasks to storage nodes and / or computing nodes according to preset conditions to complete the coal mine geological big data processing tasks; wherein the processing unit 12 includes: a resource layer for managing each node of the storage layer and the computing layer; a storage layer for analyzing the storage requirements of coal mine geological big data to meet the storage and access requirements of coal mine geological big data; a computing layer for meeting the computing and processing requirements of coal mine geological big data; a scheduling layer for scheduling and managing coal mine geological big data processing tasks to achieve efficient execution of tasks and optimal use of resources. By allocating appropriate storage nodes and / or computing nodes to the coal mine geological big data processing tasks, efficient processing of coal mine geological big data is achieved.

[0057] Optionally, in one embodiment of the present invention, the storage layer includes: a classification module, used to divide the collected coal mine data into multiple sub-datasets according to a first rule, and assign a unique identifier to each sub-dataset; a mapping module, used to establish a mapping relationship between the sub-dataset and the original sub-cluster of the storage layer according to a second rule; a processing module, used to complete distributed storage and parallel processing of the sub-dataset in the sub-cluster with the established mapping relationship; a dynamic adjustment module, used to dynamically adjust the data of the sub-dataset and the mapping relationship between the sub-dataset and the sub-cluster according to the real-time operating status of the coal mine geological big data processing system and user requests.

[0058] Optionally, in one embodiment of the present invention, the computing layer includes: a self-detection module, which is used to automatically migrate and reschedule tasks through automatic detection and failover mechanism when a computing node in the computing layer fails or fails; a dynamic adjustment module, which is used to monitor and dynamically adjust the resource utilization and load conditions of the computing nodes in the computing layer to achieve load balancing of the computing nodes and improve overall performance and resource utilization efficiency.

[0059] Optionally, in one embodiment of the present invention, the scheduling layer includes: a load balancing module, which is used to monitor the load and resource utilization of each node in the storage layer and the computing layer, and dynamically adjust and allocate computing and storage tasks to the optimal nodes through load balancing strategies; a failover module, which is used to monitor computing nodes and storage nodes in real time, and adopt fault detection and failover mechanisms to automatically migrate and reschedule tasks when a node fault or failure is detected.

[0060] like Figure 2 As shown, an embodiment of the present invention provides a general architecture diagram of a processing unit, including:

[0061] Resource layer 21 is deployed with Kubernetes (a containerized application that manages multiple hosts) resource manager. Kubernetes provides powerful resource management and task scheduling capabilities, and can intelligently allocate computing and storage tasks to different nodes in the cluster to achieve parallel processing and load balancing of tasks. Through the intelligent scheduling algorithm of the Master node, it dynamically allocates computing and storage tasks to multiple Worker nodes based on factors such as data volume, computing complexity, and performance requirements. This strategy can automatically sense changes in system load and resource utilization, and adjust task allocation in real time according to demand to achieve optimal resource utilization and maximize system performance;

[0062] The resource layer mainly builds computing, storage, and network resource pools based on virtualization technology to provide highly elastic, stable, secure, and quickly deployed virtual resources for upper-layer applications. The resource layer provides virtualized computing, storage, and network resource pools. It monitors the configuration information of the data center, the Pod in the kubernetes cluster, and the Pod's CPU, memory Mem, network Net, disk I / O, etc.

[0063] Optionally, in some embodiments, the abnormal node real-time detection and diagnosis algorithm of the system, the abnormal detection indicators CPU, Mem, I / O are calculated as follows:

[0064]

[0065]

[0066]

[0067] The meanings of the English words and letters in the above formula are as follows:

[0068]

[0069] CPU, Mem, and I / O represent the CPU usage, memory usage, and I / O load rate of the Hadoop cluster nodes, respectively. The three calculated system structure indicators are sorted in ascending order, and the indicators with usage rates below 20% and above 80% are marked as abnormal indicators.

[0070] The storage layer 22 adopts a variety of storage node deployment strategies for different types of data and access modes. The data is classified according to its nature and access frequency, and deployed on non-relational database storage nodes such as MySQL, PostgreSQL relational databases, Redis, HBase, MongoDB, Neo4J, InfluxDB, Elasticsearch, etc., to maximize the access efficiency and storage capacity of the data. At the same time, an HDFS distributed file system is deployed to store large-scale data. To ensure the security and reliability of the data, the storage layer intelligent lake warehouse also introduces data redundancy and backup mechanisms to perform redundant backup of data between different storage nodes. Once a node failure occurs, it can automatically switch to the backup node to ensure the persistence and recoverability of the data. The storage layer of this embodiment adopts a dynamic data migration strategy, which automatically migrates data from hot nodes to cold nodes according to the heat of the data and the distribution of hot and cold data to optimize the utilization efficiency of storage resources. At the same time, by monitoring the resource utilization and access status of the storage nodes, a load balancing algorithm is used to dynamically allocate data access requests to different storage nodes to achieve load balancing of the storage layer and improve the performance and availability of the entire system. In addition, the storage layer can also plan and optimize the storage layer capacity according to the data growth trend and business needs. By predicting the data growth trend, the capacity can be expanded in time to avoid the bottleneck of storage resources and ensure the scalability and stability of the system.

[0071] Optionally, in some embodiments, the storage layer provides distributed storage services, including a data warehouse and a data lake, and mainly processes the data as follows:

[0072] (1) Establish metadata and its index, including:

[0073] a) Define geoscientific metadata. Define appropriate metadata for each data type, including table name, field name, document structure, tags, and other information.

[0074] b) Establish geoscientific data index: According to the query requirements, index structured and semi-structured data to improve query performance. For unstructured data, index can be established based on file name, type, etc. In particular, for geoscientific data with spatial attributes, establish spatial coordinate index

[0075] c) Associate data with the same encoding: Establish associations based on common identifiers or key fields in the metadata so that data items with the same encoding can be associated; based on the needs of the subsystem, use relational databases (such as MySQL, PostgreSQL) or certain types of NoSQL databases (such as MongoDB) to store structured data.

[0076] (2) Data compression: Using data compression algorithms, for time series data with little change in data record values ​​of coal mine geological data (such as continuously monitored water level data), the Gorilla algorithm is used for data compression (initial time record, time difference, record value)

[0077] Δ=(t n -t n-1 )-(t n-1 -t n-2 )

[0078] In the formula, Δ is used to calculate the time difference and the recorded value respectively. The recorded value is performed using IEEE 754 floating point numbers.

[0079]

[0080] Among them, s is used to determine the sign, 0 represents positive and 1 represents negative; f represents the 11-bit significant exponent; h represents the 52-bit completed mantissa.

[0081] (2) Data storage classification: Use decision data algorithm to establish a temporary data dictionary to store the system historical data set, and compare the real-time value with the historical value to obtain the data offset value Z. Adaptive classification is performed based on the offset value. If the offset is 0, it is unknown data and is placed in a unified data lake (Hadoop HDFS); if the offset is not equal to 0, decide whether it is a state quantity. If the state quantity is greater than 1, it is placed in the Gorilla compression processing class. If the state quantity is equal to 1, it is placed in other data processing classes.

[0082] (4) Data storage sorting - cold and hot data

[0083] a) New data storage strategy: select the spatial resolution S, temporal resolution T, and data quality Q of geoscience data as indicators, and the data storage calculation data value formula is:

[0084] V(S i ,T i ,Q i )=W S ×J Si +W T ×J Ti +W Q ×J Qi

[0085] Where W S , W T , W Q are the weights of spatial resolution S, temporal resolution T, and data quality Q, which are automatically determined by the system based on the data field; J Si , J Ti , J QiThey respectively indicate whether the three indicators of data i stored in the data lake warehouse meet the application requirements. If they meet the requirements, the value is 1, and if they do not meet the requirements, the value is 0. i ,T i ,Q i ) values ​​to sort the data, with high values ​​classified as hot data and low values ​​as cold data.

[0086] b) Data sorting strategy after storage, data access heat H,

[0087]

[0088] In the formula, t i is the storage time, t is the current time, and the stored data is reordered according to the value of data access heat H.

[0089] The computing layer 23 is deployed with the Hadoop (distributed system infrastructure) ecosystem and the Spark computing engine. The Spark computing engine can read data directly from the HDFS distributed file system and execute computing tasks on Kubernetes, providing efficient data processing and analysis capabilities. At the same time, the Flink distributed computing framework is also deployed to realize streaming data processing and batch data processing. In order to cope with changes in system load and fluctuations in demand, the computing layer supports the elastic expansion and contraction of the system. According to actual needs, the number of computing nodes is automatically increased or decreased to achieve elastic scaling of computing resources to ensure the high availability and stability of the system. At the same time, in order to improve the fault tolerance of the system, this embodiment also introduces a fault detection and failover mechanism to monitor the computing nodes in real time. Once a node failure or failure is found, task migration and rescheduling are automatically performed to ensure the stability and reliability of the system. In addition, the computing layer can also monitor the resource utilization and task completion of the computing nodes, and use a load balancing algorithm to dynamically allocate computing tasks to different nodes to achieve load balancing of the computing layer and improve the performance and availability of the entire system.

[0090] Optionally, in some embodiments, in order to be compatible with the existing coal mine system technical requirements, the computing layer needs to be compatible with the computing tasks of Hadoop, Spark, Flink and other algorithms, and the algorithm upgrades and the smooth upgrade of the version;

[0091] The computing layer adopts distributed computing. According to the coal mine geological data computing needs, a large number of computing tasks are split into smaller computing units, and the computing tasks are allocated to different nodes for computing, so as to improve computing efficiency.

[0092] In some embodiments, the computing layer may also process the data as follows:

[0093] (1) Preprocessing of coal mine geological data: There may be missing values ​​in coal mine geological data during collection, which need to be filled or processed by appropriate methods. Common methods include interpolation (linear interpolation, polynomial interpolation) and using generative adversarial networks (GANs) to generate missing values.

[0094] (2) Integration and upgrading of coal mine seismic exploration data processing algorithms, including static correction, pre-stack denoising, deconvolution, velocity analysis, dynamic correction stacking analysis (NMO), migration imaging, etc.; for example, coal mine seismic exploration data denoising algorithm: seismic data inevitably contain noise, such as random noise, abnormal noise, surface waves, multiple waves, etc. These noises will interfere with the imaging inversion effect of seismic data. Common denoising algorithms are: ① Convolutional neural network (CNN) uses the noise in synthetic data as a label during the model training period through CNN, so that the model learns to identify the noise in the data and suppresses random noise, linear noise, and multiple waves; ② Generative Adversarial Networks (GANs) suppress seismic random noise; ③ U-Net based on deep learning removes random noise, and the U-Net network is added to the residual block to enhance the suppression of random noise and multiple waves. The mathematical model for simultaneous reconstruction and denoising using the U-Net network can be expressed as:

[0095] Y = Net(X, θ)

[0096] Where X and Y represent input and output data, respectively. θ represents the network parameters that need to be trained during network training, and Net(·) represents U-Net. Find a suitable network parameter θ so that after inputting noisy seismic data X, noise-free seismic data Y is output.

[0097] The scheduling layer 24 is deployed with Zookeeper (distributed application coordination service) as a coordination service for task scheduling and management. Zookeeper provides a highly available distributed coordination mechanism for monitoring task status, managing task queues and node registration information, etc., to ensure the smooth execution of tasks and the stability of the system. Through the Zookeeper coordination service, a dynamic task scheduling strategy is implemented. According to factors such as task type, priority and resource requirements, tasks are dynamically assigned to the most suitable nodes in the cluster to achieve efficient execution of tasks and optimal utilization of resources. In order to cope with changes in system load and fluctuations in demand, the scheduling layer supports elastic expansion and contraction of task scheduling. According to actual needs, the number of task execution nodes is automatically increased or decreased to achieve elastic scaling of task scheduling to ensure high availability and stability of the system. At the same time, in order to improve the fault tolerance of the system, this embodiment also introduces a fault detection and failover mechanism. Zookeeper monitors the status of the task execution node, and once a node failure or failure is found, it automatically migrates and reschedules tasks to ensure continuous execution of tasks and stable operation of the system. According to the feedback of task execution and system performance, the task scheduling strategy is continuously optimized. By real-time monitoring of task execution time, resource utilization and other indicators, the task scheduling strategy can be adjusted to improve task execution efficiency and the overall performance of the system.

[0098] The scheduling layer can be a cloud computing task scheduling model, which can be abstracted as transmitting a large number of tasks to the data center. Subsequently, the data center assigns the work in the task concentration to independent virtual nodes in the resource pool for calculation based on resource allocation and task scheduling strategies. The optimization of task scheduling strategies is considered from the following aspects: economic benefits, QoS requirements (including system bandwidth, computing and storage capacity and other parameters, improving these indicators), task response time requirements (single task response time, total task response time, average response time), fairness, fault tolerance (coal mine geoscience big data system services are provided by multiple node machines, and conflicts and data transmission failures are inevitable. Good fault tolerance is the guarantee of providing reliable services and ensuring the stability of the system), load balancing.

[0099] Optionally, in some embodiments, the scheduling layer can comprehensively analyze the length of the system's jobs, taking into account the order in which the jobs arrive, so that long jobs will not be left without service for a long time. The system adopts a high response ratio priority scheduling algorithm. When job scheduling is required, the algorithm will select high-priority jobs to enter the memory. The multi-threaded technology based on the high response ratio priority scheduling algorithm calculates data. While reading and writing operations on the system storage layer database, the computing node CPU of the computing layer is used to calculate the acquired data, which can effectively reduce the program blocking time, improve CPU utilization, and thus improve computing efficiency. Response Ratio R P= Response time / Service request time,

[0100]

[0101] Among them, O(x) is the time complexity required for system job score calculation or database operation, The sum of the time complexity of threads that have been processed since a thread joined the thread pool.

[0102] like Figure 3 As shown, it is a resource layer architecture composition diagram provided by an embodiment of the present invention, with Kubernetes cluster as the basis of the system resource layer, and the architecture composition includes: 31. Application program interface service, which provides the only entrance for resource operation, and provides authentication, authorization, access control, API registration and discovery mechanisms; 32. Controller (controller manager), responsible for maintaining the status of the cluster, such as fault detection, automatic expansion, rolling update, etc.; 33. Scheduler (scheduler), responsible for scheduling resources, and scheduling Pods to corresponding machines according to predetermined scheduling strategies; 34. Kubelet, responsible for maintaining the life cycle of containers, and also responsible for the management of Volume (CVI) and network (CNI); 35. Proxy, responsible for providing service discovery and load balancing within the cluster for Service.

[0103] like Figure 4 As shown, a resource layer deployment flow chart provided by an embodiment of the present invention is used to build a Kubernetes cluster as the basis of the system resource layer. The process includes: S41, data center hardware and network preparation, nodes can communicate with each other, ensure that the external network can access the cluster, and all machines uniformly use the Linux operating system, specifically the Ubuntu operating system. The minimum configuration information of the resources allocated to each node should meet the requirements of 2GB or more memory, 2 cores or more CPU, and 50GB or more hard disk; S42, all nodes install Docker and Kubernetes components, including Kubeadm, Kubelet and kubectl, etc.; S43, Kubeadm boots the cluster and configures the Kubernetes image source; S44, initialize the Master node, add the Worker node, and add the Master node domain name mapping to all machines; S45, deploy the monitoring system, network plug-in, storage plug-in, Ingress controller, log collection tool, and deploy and install the Dashboard on the monitoring node machine to achieve visual management of the Kubernetes cluster; S46, make each component into a Docker container image to ensure the portability and scalability of the system, and facilitate deployment and scheduling in the Kubernetes cluster.

[0104] Furthermore, in one embodiment of the present invention, a resource object such as Deployment or StatefulSet of Kubernetes is used to deploy the container image to the Worker node of the cluster. According to actual needs, multiple Deployments or StatefulSets can be created to deploy different system components and services respectively.

[0105] Furthermore, in one embodiment of the present invention, the Service resource object of Kubernetes is used to expose the service to the outside of the cluster to achieve external access. You can choose a Service of ClusterIP, NodePort or LoadBalancer type and configure it according to load balancing and access control requirements.

[0106] Furthermore, in one embodiment of the present invention, Kubernetes's Persistent Volume (Chinese name: Persistent Volume, PV for short) and Persistent Volume Claim (Chinese name: Persistent Volume Claim, PVC for short) resource objects are used to achieve dynamic management of storage. According to storage requirements, PV and PVC are created for different services and bound to the corresponding Pods to ensure data persistence and high availability. You can choose a Kubernetes distribution suitable for the coal mine geological big data Zhihu Warehouse system, such as OpenShift, Rancher, or configure a native Kubernetes cluster yourself. Make sure that the cluster contains a Master node and multiple Worker nodes to achieve task management and distributed computing.

[0107] Furthermore, in one embodiment of the present invention, configuration management uses Kubernetes' ConfigMap and Secret resource objects to store configuration information and sensitive data (such as passwords and keys) as Kubernetes resources. This ensures the flexibility and security of the configuration and facilitates management and updating in the cluster.

[0108] Furthermore, in one embodiment of the present invention, the Horizontal PodAutoscaler (HPA) resource object of Kubernetes is used to realize automatic horizontal expansion and elastic scaling of the number of Pods according to CPU utilization or custom indicators. In this way, the capacity of the system can be automatically adjusted according to the actual load conditions to ensure high performance and high availability of the system.

[0109] The embodiment of the present invention also provides a method for processing coal mine geological big data, such as Figure 1As shown, it includes: S51, receiving a big data processing task; S52, allocating the big data processing task to storage nodes and / or computing nodes according to preset conditions to complete the big data processing task.

[0110] The step of allocating the big data processing task to storage nodes and / or computing nodes according to preset conditions to complete the big data processing task includes:

[0111] Manage the storage nodes for coal mine geological big data and the computing nodes for computing and processing big data;

[0112] Analyze the storage requirements of coal mine geological big data, and store and access coal mine geological big data;

[0113] Calculate and process coal mine geological big data;

[0114] Schedule and manage coal mine geological big data processing tasks to achieve efficient task execution and optimal utilization of resources.

[0115] Optionally, in one embodiment of the present invention, the analyzing storage requirements of coal mine geological big data, storing and accessing the coal mine geological big data, includes:

[0116] Dividing the collected coal mine data into a plurality of sub-data sets according to a first rule, and assigning a unique identifier to each sub-data set;

[0117] Establishing a mapping relationship between the sub-dataset and the original sub-cluster of the storage layer according to the second rule;

[0118] The sub-datasets are distributedly stored and processed in parallel in the sub-clusters that establish mapping relationships.

[0119] According to the real-time operation status of the coal mine geological big data processing system and the user request situation, the data of the sub-datasets and the mapping relationship between the sub-datasets and the sub-clusters are dynamically adjusted.

[0120] Optionally, in one embodiment of the present invention, the calculating and processing of coal mine geological big data includes:

[0121] When a computing node fails or fails, tasks are automatically migrated and rescheduled through automatic detection and failover mechanisms;

[0122] Monitor and dynamically adjust the resource utilization and load conditions of computing nodes to achieve load balancing of computing nodes and improve overall performance and resource utilization efficiency.

[0123] Optionally, in one embodiment of the present invention, the scheduling and management of coal mine geological big data processing tasks to achieve efficient execution of tasks and optimal utilization of resources includes:

[0124] Monitor the load and resource utilization of computing nodes and storage nodes, and dynamically adjust and allocate computing and storage tasks to the optimal nodes through load balancing strategies;

[0125] The computing nodes and storage nodes are monitored in real time, and fault detection and failover mechanisms are adopted. When node failure or inefficiency is detected, tasks are automatically migrated and rescheduled.

[0126] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0127] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referenced to each other, and each embodiment focuses on the differences from other embodiments.

[0128] In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0129] For the convenience of description, the above device is described by dividing the functions into various units / modules. Of course, when implementing the present invention, the functions of each unit / module can be implemented in the same or multiple software and / or hardware.

[0130] A person skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. The storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0131] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed by the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A coal mine geological big data processing system, characterized in that: include: A receiving unit, used for receiving coal mine geoscience big data processing tasks; A processing unit, used for allocating the coal mine geoscience big data processing task to storage nodes and / or computing nodes according to preset conditions to complete the coal mine geoscience big data processing task; Wherein, the processing unit comprises: The resource layer is used to manage the nodes of the storage layer and computing layer; The storage layer is used to analyze the storage requirements of coal mine geological big data tasks and meet the storage and access requirements of coal mine geological big data; The computing layer is used to meet the computing and processing requirements of coal mine geoscience big data tasks; The scheduling layer is used for scheduling and management of coal mine geological big data processing tasks to achieve efficient execution of tasks and optimal utilization of resources; The storage layer is specifically used to adopt a variety of storage node deployment strategies for different types of data and access modes; classify data according to its nature and access frequency, and deploy them on MySQL and PostgreSQL relational databases, as well as Redis, HBase, MongoDB, Neo4J, InfluxDB and Elasticsearch non-relational database storage nodes, and deploy the HDFS distributed file system at the same time to store large-scale data; introduce data redundancy and backup mechanisms to perform redundant backup of data between different storage nodes, and automatically switch to the backup node when a node fails; adopt a dynamic data migration strategy to automatically migrate data from hot nodes to cold nodes according to the popularity of data and the distribution of hot and cold data, so as to optimize the utilization efficiency of storage resources; perform capacity planning and optimization of the storage layer according to the data growth trend and business needs; and timely expand capacity by predicting the growth trend of data to avoid bottlenecks in storage resources; The storage layer is also used for: Establish metadata and its index, including: define geoscientific metadata: define appropriate metadata for each data type, including table name, field name, document structure and tag information; establish geoscientific data index, and establish indexes for structured and semi-structured data according to query requirements to improve query performance; for unstructured data, establish indexes according to file name and type; among them, for geoscientific data with spatial attributes, establish spatial coordinate indexes; Establish associations based on common identifiers or key fields in metadata so that data items with the same encoding can be associated; use certain types of relational databases or NoSQL databases to store structured data based on the needs of the subsystem; Using a data compression algorithm, for time-series data with little change in the recorded values of coal mine geoscience data, the Gorilla algorithm is used for data compression. Calculate the time difference and recorded value of the data. The recorded value is in IEEE 754 floating-point format, (0 < e < 2047); where s is used to determine the sign, 0 for positive and 1 for negative; f represents the 11-bit exponent of the significant digits; h represents the 52-bit mantissa with padding; Adopt decision data algorithm, establish temporary data dictionary, store system historical data set, and obtain data offset value Z by comparing real-time value and historical value. Adaptive classification is performed according to offset value Z. If the offset is 0, it is unknown data and put into the unified data lake Hadoop HDFS. If the offset is not equal to 0, decide whether it is a state quantity. If the state quantity is greater than 1, put it into the Gorilla compression processing class. If the state quantity is equal to 1, put it into other data processing classes. According to the data value formula, the data is sorted into cold and hot data storage, including: selecting the spatial resolution S, temporal resolution T, and data quality Q of geoscience data as indicators, and calculating the data value formula as follows: ; In the formula, V(S i ,T i ,Q i ) is the data value, W S , W T , W Q are the weights of spatial resolution S, temporal resolution T, and data quality Q, which are automatically determined by the system based on the data field; J Si , J Ti , J Qi Respectively indicate whether the three indicators of data i stored in the data lake warehouse meet the application requirements. If they meet the requirements, the value is 1, and if they do not meet the requirements, the value is 0; The stored data is reordered using the data access heat formula; the data access heat formula is: ； Where H is the data access heat, t i is the storage time, t is the current time, V(S i ,T i ,Q i ) is the data value, and the stored data is reordered according to the value of data access heat H.

2. The coal mine geoscience big data processing system according to claim 1, characterized in that: The storage layer includes: A classification module, used for dividing the collected coal mine data into a plurality of sub-data sets according to a first rule, and assigning a unique identifier to each sub-data set; A mapping module, used to establish a mapping relationship between the sub-dataset and the original sub-cluster of the storage layer according to the second rule; A processing module, used for completing distributed storage and parallel processing of the sub-datasets in the sub-clusters with established mapping relationships; The dynamic adjustment module is used to dynamically adjust the data of the sub-datasets and the mapping relationship between the sub-datasets and the sub-clusters according to the real-time operating status of the coal mine geological big data processing system and user requests.

3. The coal mine geoscience big data processing system according to claim 1, characterized in that: The computing layer includes: The self-detection module is used to automatically migrate and reschedule tasks through automatic detection and failover mechanisms when computing nodes in the computing layer fail or fail; The dynamic adjustment module is used to monitor and dynamically adjust the resource utilization and load conditions of the computing nodes in the computing layer to achieve load balancing of the computing nodes and improve overall performance and resource utilization efficiency.

4. The coal mine geoscience big data processing system according to claim 1, characterized in that: The scheduling layer includes: The load balancing module is used to monitor the load and resource utilization of each node in the storage layer and computing layer, and dynamically adjust and allocate computing and storage tasks to the optimal node through load balancing strategies; The failover module is used to monitor computing nodes and storage nodes in real time, and adopts fault detection and failover mechanisms. When a node failure or failure is found, it automatically migrates and reschedules tasks.

5. A method for processing coal mine geological big data, characterized in that: include: Accept coal mine geological big data processing tasks; Allocating the coal mine geoscience big data processing task to storage nodes and / or computing nodes according to preset conditions to complete the coal mine geoscience big data processing task; The method of allocating the coal mine geoscience big data processing task to storage nodes and / or computing nodes according to preset conditions to complete the coal mine geoscience big data processing task includes: Manage the storage nodes for coal mine geological big data and the computing nodes for computing and processing big data; Analyze the storage requirements of coal mine geological big data, and store and access coal mine geological big data; Calculate and process coal mine geological big data; Schedule and manage coal mine geological big data processing tasks to achieve efficient execution of tasks and optimal use of resources; The storage requirements of coal mine geological big data are analyzed, and the coal mine geological big data is stored and accessed, including: In response to different types of data and access patterns, a variety of storage node deployment strategies are adopted to classify data according to its nature and access frequency, and deploy them on MySQL and PostgreSQL relational databases, as well as Redis, HBase, MongoDB, Neo4J, InfluxDB and Elasticsearch non-relational database storage nodes. At the same time, the HDFS distributed file system is deployed to store large-scale data; Introducing data redundancy and backup mechanisms to perform redundant backup of data between different storage nodes. When a node fails, it automatically switches to the backup node. Adopt dynamic data migration strategy to automatically migrate data from hot nodes to cold nodes according to the popularity of data and the distribution of hot and cold data, so as to optimize the utilization efficiency of storage resources; Carry out capacity planning and optimization of the storage layer according to data growth trends and business needs; timely expand capacity by predicting data growth trends to avoid storage resource bottlenecks; The analysis of the storage requirements of coal mine geological big data, and the storage and access to the coal mine geological big data also include: Establish metadata and its index, including: define geoscientific metadata: define appropriate metadata for each data type, including table name, field name, document structure and tag information; establish geoscientific data index, and establish indexes for structured and semi-structured data according to query requirements to improve query performance; for unstructured data, establish indexes according to file name and type; among them, for geoscientific data with spatial attributes, establish spatial coordinate indexes; Establish associations based on common identifiers or key fields in metadata so that data items with the same encoding can be associated; use data types in relational databases or NoSQL databases to store structured data based on the needs of the subsystem; Using a data compression algorithm, for time series data with insignificant changes in the recorded values of coal mine geoscience data, the Gorilla algorithm is used for data compression. Calculate the time difference and recorded value of the data. The recorded value is in IEEE 754 floating-point number format, (0 < e < 2047); where s is used to determine the sign, 0 for positive and 1 for negative; f represents the 11-bit exponent of the significant digits; h represents the 52-bit mantissa with padding; Adopt decision data algorithm, establish temporary data dictionary, store system historical data set, and obtain data offset value Z by comparing real-time value and historical value. Adaptive classification is performed according to offset value Z. If the offset is 0, it is unknown data and put into the unified data lake Hadoop HDFS. If the offset is not equal to 0, decide whether it is a state quantity. If the state quantity is greater than 1, put it into the Gorilla compression processing class. If the state quantity is equal to 1, put it into other data processing classes. According to the data value formula, the data is sorted into cold and hot data storage, including: selecting the spatial resolution S, temporal resolution T, and data quality Q of geoscience data as indicators, and calculating the data value formula as follows: ; In the formula, V(S i ,T i ,Q i ) is the data value, W S , W T , W Q are the weights of spatial resolution S, temporal resolution T, and data quality Q, which are automatically determined by the system based on the data field; J Si , J Ti , J Qi Respectively indicate whether the three indicators of data i stored in the data lake warehouse meet the application requirements. If they meet the requirements, the value is 1, and if they do not meet the requirements, the value is 0; The stored data is reordered using the data access heat formula; the data access heat formula is: ； Where H is the data access heat, t i is the storage time, t is the current time, V(S i ,T i ,Q i ) is the data value, and the stored data is reordered according to the value of data access heat H.

6. The method for processing coal mine geological big data according to claim 5, characterized in that: The storage requirements of coal mine geological big data are analyzed, and the coal mine geological big data is stored and accessed, including: Dividing the collected coal mine data into a plurality of sub-data sets according to a first rule, and assigning a unique identifier to each sub-data set; Establishing a mapping relationship between the sub-dataset and the original sub-cluster of the storage layer according to the second rule; The sub-datasets are distributedly stored and processed in parallel in the sub-clusters with which the mapping relationship is established; According to the real-time operation status of the coal mine geological big data processing system and the user request situation, the data of the sub-datasets and the mapping relationship between the sub-datasets and the sub-clusters are dynamically adjusted.

7. The method for processing coal mine geological big data according to claim 5, characterized in that: The calculation and processing of coal mine geological big data includes: When a computing node fails or fails, tasks are automatically migrated and rescheduled through automatic detection and failover mechanisms; Monitor and dynamically adjust the resource utilization and load conditions of computing nodes to achieve load balancing of computing nodes and improve overall performance and resource utilization efficiency.

8. The method for processing coal mine geological big data according to claim 5, characterized in that: The scheduling and management of coal mine geological big data processing tasks to achieve efficient execution of tasks and optimal utilization of resources include: Monitor the load and resource utilization of computing nodes and storage nodes, and dynamically adjust and allocate computing and storage tasks to the optimal nodes through load balancing strategies; The computing nodes and storage nodes are monitored in real time, and fault detection and failover mechanisms are adopted. When node faults or failures are detected, tasks are automatically migrated and rescheduled.

Images