Streaming data and mass batch data storage method

Abstract

The invention relates to a streaming data and mass batch data storage method, which belongs to the technical field of data storage, and comprises the following steps: classifying mine data into streaming data and batch data; according to the method, a mixed storage architecture is constructed, the mixed storage architecture comprises a data acquisition layer, a batch processing layer, a speed layer, a service layer and an application layer, so that streaming data and batch data are processed and stored by adopting different rules, and the method comprises the following steps of: acquiring the streaming data by adopting an acquisition adaptation gateway and a real-time message queue Kafka; collecting the batch data by adopting a relational database DM8 and a timing scheduling Airflow mode; the batch processing layer is used for dividing the data into cold and hot temperature data and performing classified storage on the data; the speed layer is used for processing and storing the real-time streaming data; the service layer is used for carrying out data integration on the temporary stream processing data and the batch processing result data and providing data query interface service; and the application layer is used for setting a business application module and accessing each database to realize data application.

Description

Technical Field

[0001] The present invention belongs to the technical field of data storage and relates to a method for storing streaming data and massive batch data. Background Art

[0002] During the process of intelligent mining and safe production management in mines, a large amount of streaming data and massive batch data will be generated.

[0003] In terms of streaming data, intelligent mining primarily focuses on monitoring various environmental parameters within the mine, the location of personnel and vehicles, slope monitoring, tailings ponds, and real-time online monitoring of shearers, conveyors, ventilators, and various auxiliary production equipment. This streaming data monitoring primarily comes from various sensors and industrial control intelligent instruments, such as sensors for monitoring various underground environmental parameters, including methane, carbon monoxide, temperature, wind speed, smoke, pressure, and start / stop sensors. This data is highly real-time, continuously generated, and dynamically changing in volume, requiring rapid, real-time collection, transmission, and storage to enable real-time monitoring of the mine environment and equipment operating status.

[0004] In terms of massive batch data, it includes historical data on mining, equipment maintenance records, equipment failure records, mine safety production report ledgers, mine safety risk hazard records, mine high-risk operation lists, etc. These data have been accumulated over a long period of time to form a large-scale data set, which has important reference value for decision-making analysis such as mine production optimization, equipment fault diagnosis and prediction, and safety risk assessment.

[0005] Traditional data storage methods struggle to simultaneously meet the real-time storage and processing needs of streaming data, as well as the low-cost, efficient storage and rapid query requirements of massive batch data, making it impossible to achieve comprehensive and efficient management of both types of data. On the one hand, the complex underground environment of underground mines, characterized by high humidity, high dust levels, and strong electromagnetic interference, places extremely high demands on the stability and reliability of data storage equipment. On the other hand, existing storage technologies suffer from significant performance deficiencies and high costs when handling the high-concurrency writes and real-time queries of mining streaming data, as well as the efficient storage and complex analysis of massive batch data. For example, using a single relational database to store mining data is unable to cope with the high-concurrency writes of streaming data and is inefficient when processing complex queries of massive batch data. Simply storing data in a standard file system makes it difficult to achieve effective data management and rapid retrieval.

[0006] Therefore, there is an urgent need for an efficient storage method for streaming data and massive batch data specifically for mining scenarios to ensure safe production and intelligent development of coal mines and non-coal mines. Summary of the Invention

[0007] In view of this, an object of the present invention is to provide a method for storing streaming data and massive batch data.

[0008] In order to achieve the above object, the present invention provides the following technical solutions:

[0009] A method for storing streaming data and massive batch data, comprising:

[0010] Classify mining data into streaming data and batch data;

[0011] Build a hybrid storage architecture, including data collection layer, batch processing layer, speed layer, service layer, and application layer, so that streaming data and batch data are processed and stored using different rules. Specifically,

[0012] Streaming data is collected using a collection adapter gateway and the real-time message queue Kafka; batch data is collected using the relational database DM8 and scheduled Airflow.

[0013] The batch processing layer is used to divide data into cold data, hot data, and warm data, and store the data in a classified manner;

[0014] The speed layer is used to process and store real-time streaming data;

[0015] The service layer is used to integrate temporary stream processing data and batch processing result data, and provide data query interface services;

[0016] The application layer is used to set up business application modules and access various databases to implement data applications.

[0017] Furthermore, the mine data is classified into stream data and batch data, specifically including:

[0018] A data acquisition and reception program is deployed on the mine side to receive real-time data from underground sensors and batch data submitted by the surface system. The data is classified using a recognition algorithm based on mine data features:

[0019] If the data contains continuous timestamps and is transmitted in real-time stream form, it is determined to be coal mine streaming data;

[0020] If the data is submitted in batches and the data volume is large, it is determined to be coal mine batch data.

[0021] Furthermore, the streaming data includes environmental monitoring, personnel and vehicle positioning, slope monitoring, tailings pond monitoring, and equipment operating condition data; the batch data includes report and ledger data;

[0022] The environmental monitoring data includes methane, carbon monoxide, wind speed, temperature, and equipment start and stop, and is collected through databases, web APIs, and message queues.

[0023] The personnel and vehicle positioning data includes personnel information, personnel real-time location, personnel trajectory, vehicle information and vehicle location, and is collected through databases and WEB APIs;

[0024] The slope monitoring data includes slope displacement, stress, and groundwater;

[0025] The tailings pond monitoring data includes tailings pond water level, rainfall, dry beach, infiltration line, displacement, and video;

[0026] The equipment operating condition data includes the equipment's operating status, motor temperature, voltage, current, and power;

[0027] The report ledger data includes production output or footage ledger, equipment maintenance and equipment failure records.

[0028] Furthermore, the batch processing layer divides the data into hot data, warm data and cold data according to the access frequency and time attributes of the mine data, uses HBase to store cold data, relational database DM8 to store warm data, and Redis memory database to store hot data.

[0029] Furthermore, the batch processing layer adopts different partitioning rules for different data types, including:

[0030] For equipment operating condition data, partition them according to the equipment's subsystem ID and data type;

[0031] Partition production data by shift, mine or ore type;

[0032] Spark Batch is used to process offline tasks, including monthly production statistics and equipment failure rate analysis.

[0033] Furthermore, the speed layer collects real-time streaming data from sensors, transmits it through Kafka message queues by topic partitions, processes it through Spark Streaming, and finally parses the streaming data by the real-time parsing module. If the real-time value exceeds the set alarm threshold, the alarm mechanism is triggered and the data is written to the alarm center. If the data is normal, the latest sampled real-time data is written to the Redis in-memory database.

[0034] Aggregate device status based on a fixed time window and perform statistics on the data within the time window, including the maximum minute value, maximum time, minimum value, minimum time, average value, and highest level status of the monitoring point;

[0035] For discrete quantities, calculate and count the state flip records of monitoring points.

[0036] Furthermore, consider fault-tolerant strategies, using Kafka cluster partition replicas to cope with network disconnection or congestion.

[0037] Furthermore, data integration at the service layer includes:

[0038] For real-time device status, Redis stores the latest 5-minute data for temporary stream processing, and HBase stores the full historical data for batch processing results. The integration method prioritizes stream results for real-time queries and periodically uses batch results for corrections.

[0039] For statistical data, the temporary data for stream processing is the hourly estimated value of stream computing, and the batch processing result data is the precise value at the end of the batch processing day. The integration method is to overwrite the stream results with the batch results at 24:00 every day.

[0040] Furthermore, the data query interface services of the service layer are divided into real-time query and historical query:

[0041] Real-time queries include real-time device location retrieval in Elasticsearch and hot data cache in Redis;

[0042] Historical queries use the HBase SQL interface.

[0043] Furthermore, the application layer includes the following applications:

[0044] Safety monitoring: gas over-limit warning, underground personnel positioning and tracking;

[0045] Production optimization: ore sorting parameter adjustment, equipment predictive maintenance;

[0046] Management decisions: resource reserve analysis, production cost accounting reports.

[0047] The beneficial effects of the present invention are as follows: the present invention protects a method for efficient storage of streaming data and massive batch data, and divides the real-time data collected by IoT devices in coal mines and non-coal mines into three categories of data: cold, hot and warm, according to the characteristics of the data hotspot cycle, and performs data grading and classification and distributed storage according to different themes, thereby realizing hot and cold data partitioning, convenient access and efficient storage. The present invention comprehensively utilizes big data processing and storage technology and deeply integrates it with modern mine safety online monitoring, and uniformly collects, efficiently stores and visualizes online monitoring data such as environmental monitoring, personnel and vehicle monitoring, and major electromechanical equipment monitoring related to coal mines and non-coal mines, ultimately realizing real-time and efficient storage and processing of mine streaming data, as well as low-cost, highly reliable storage and fast query of massive batch data, thereby improving mine data monitoring capabilities and assisting mine safety production.

[0048] Other advantages, objects, and features of the present invention will be described in part in the following description and, in part, will be apparent to those skilled in the art upon examination of the following description or may be learned from practice of the present invention. The objects and other advantages of the present invention may be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be described in detail below with reference to the accompanying drawings, in which:

[0050] Figure 1 This is the flow chart of speed layer processing. DETAILED DESCRIPTION

[0051] The following describes the embodiments of the present invention by means of specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present invention, and the following embodiments and features in the embodiments can be combined with each other without conflict.

[0052] It should be noted that the illustrations provided in the following embodiments are merely schematic illustrations of the basic concept of the present invention. Therefore, the illustrations only show components related to the present invention and are not drawn according to the number, shape, and size of components in actual implementation. In actual implementation, the type, quantity, and proportion of each component may be changed arbitrarily, and the component layout may also be more complex.

[0053] In the following description, numerous details are discussed to provide a more thorough explanation of the embodiments of the present invention. However, it will be apparent to those skilled in the art that the embodiments of the present invention may be practiced without these specific details. In other embodiments, well-known structures and devices are shown in block diagram form rather than in detail to avoid obscuring the embodiments of the present invention.

[0054] Example 1:

[0055] The present invention provides an efficient storage method for streaming data and massive batch data applicable to coal mine and non-coal mine scenarios. The Kafka distributed message queue is used as a streaming data buffer to provide a stable data source for subsequent processing. The in-memory database Redis is used to efficiently store each sampling data of the mine Internet of Things equipment. The relational database DM8 is used to perform structured storage of medium- and short-term data. The streaming data is optimized and stored through the HBase time series database. By combining multiple databases, the goal of hierarchical and classified efficient storage of mine streaming data and massive batch data is ultimately achieved.

[0056] First, the data information is categorized. Mine data is primarily divided into streaming data such as environmental monitoring, personnel and vehicle location, slope monitoring, tailings pond monitoring, and equipment operating conditions, as well as batch data such as report and ledger data. Different data types require different data collection methods.

[0057] A data collection and reception program is deployed at the mine site to receive real-time data from underground sensors and batch data submitted by the surface system. Data is classified using a recognition algorithm based on mine data characteristics. If the data contains continuous timestamps and is transmitted in real-time stream form, such as real-time gas concentration monitoring data or real-time equipment operating status data, it is considered coal mine streaming data. If the data is submitted in batches and is large in volume, such as monthly coal mining output data or annual equipment maintenance records, it is considered coal mine massive batch data.

[0058] Environmental monitoring data: Most underground and open-pit mines have safety monitoring systems installed to monitor environmental parameters in underground workplaces, including methane, carbon monoxide, wind speed, temperature, and equipment startup and shutdown. Therefore, all monitoring data only needs to be collected from the safety monitoring subsystem. Data collection methods support at least databases, web APIs, and message queues.

[0059] Personnel and vehicle positioning data: Most underground and open-pit mines have installed personnel positioning systems, and those with vehicle positioning requirements also have these systems installed. Therefore, only the personnel and vehicle positioning subsystems need to collect positioning data. These systems must collect at least personnel information, real-time personnel location, personnel trajectory, and vehicle information and location data. Data collection methods must support databases, web APIs, and other methods.

[0060] Slope monitoring data: For most open-pit coal and non-coal mines, slope monitoring is a crucial monitoring subsystem. Slope monitoring data, including slope displacement, stress, and groundwater levels, can be used to predict the potential for slope instability and landslide risk, providing a foundation for preventing and controlling landslides and potential sliding and creep deformation.

[0061] Tailings pond monitoring data: For tailings pond monitoring, monitoring data typically includes tailings pond water level, rainfall, dry beach, infiltration line, displacement, and video. Compared to monitoring data from mining environmental monitoring sensors, data generation and transmission frequency is slightly longer. By analyzing and processing information on deformation factors, related factors, and inducing factors, the stability and changing trends of the disaster deformation can be determined and early warnings can be issued.

[0062] Equipment operating condition data: Mines have a large number of major equipment, including coal mining machines, roadheaders, main ventilation fans, air compressors, main drainage pumps, belt conveyors, hoists, and nitrogen generators. Non-coal mines include ventilation fans, hoists, conveyors, filling equipment, and sewage discharge pumps. Key monitoring parameters include equipment operating status, motor temperature, voltage, current, and power. Data collection is typically performed using specialized industrial control software, such as WinCC and Force Control, or through PLC control cabinets. These specialized industrial control software allows for the direct collection of equipment operating condition data from these data sources. This operating condition data typically includes monitoring values, timestamps, and data quality information. Supported data sources include PLCs, OPC, and Modbus.

[0063] Report ledger data: Generally, mine ledger data includes production output or footage records, equipment maintenance and equipment failure records, etc. The characteristic of this data is that it is statistically analyzed based on a certain time period.

[0064] The second step is to build a hybrid storage architecture. Because this architecture needs to process both streaming and batch data from mining operations, a hybrid storage solution is adopted. In mining scenarios, the hybrid storage architecture can be divided into five logical layers, forming a complete closed loop from data collection to application services.

[0065] Data collection layer:

[0066] The data collection layer collects and accesses multi-source heterogeneous data based on the characteristics of mining data. The data source formats are different for different data objects.

[0067] (1) Data source

[0068] Streaming data: 1) Underground sensors (gas concentration, mine pressure, equipment vibration), IoT devices (personnel and vehicle positioning, conveyor belt speed), etc., the data source is generally FTP, WEB API or database; 2) Operating condition data of industrial control equipment (such as motor voltage, current, bearing temperature, etc.), the data source is generally PLC, OPC or Modbus, etc.

[0069] For batch data: 1) Mine production report ledgers, generally in PDF, Excel, or CSV format; 2) Equipment maintenance records or equipment failure records, generally in the form of structured forms as the data source.

[0070] (2) Collection technology

[0071] Streaming collection: Collection adapter gateway + Kafka (real-time message queue), supporting edge node deployment (such as explosion-proof gateway).

[0072] Batch collection: DM8 (relational database) + Airflow (scheduled scheduling), connecting to historical data of ERP and MES systems.

[0073] Batch layer:

[0074] The batch processing layer's primary function is to store and compute historical data. It primarily uses HBase to store historical sampling data, the relational database DM8 to store interim data, and the Redis in-memory database to store the latest sampling data. This categorized storage of hot, cold, and warm data significantly improves data storage and access efficiency.

[0075] Different partitioning rules are used for different data types:

[0076] Equipment operating condition data: partitioned by the equipment's subsystem ID and data type;

[0077] Production data: partitioned by shift, mine or ore type.

[0078] Spark Batch is used to process offline tasks such as monthly production statistics and equipment failure rate analysis.

[0079] Velocity Layer:

[0080] The speed layer focuses on processing real-time stream data. The logic diagram of the real-time stream processing link is as follows: Figure 1 shown.

[0081] The real-time streaming data collected by sensors is transmitted through the Kafka message queue by topic partition, processed by Spark Streaming, and finally parsed by the real-time parsing module. If the real-time value exceeds the set alarm threshold, the alarm mechanism is triggered and the data is written to the alarm center. If the data is normal, the latest sampled real-time data is written to the Redis in-memory database.

[0082] At the same time, device status needs to be aggregated based on fixed time windows. Typically, statistics are collected for data within a 1-minute or 5-minute data segment, including the maximum minute value, maximum time, minimum value, minimum time, average value, and highest level status of the monitoring point. For discrete quantities, state rollover records (i.e., value change and state change records) need to be calculated and counted.

[0083] The architecture also takes into account fault-tolerance strategies, namely, using Kafka cluster partition replicas to deal with network disconnection or congestion (for example, caching data to edge nodes when the network is disconnected).

[0084] Service Layer

[0085] The service layer focuses on data integration and provides data query interface services.

[0086] Table 1 shows the data merging strategies for some typical scenarios.

[0087] Table 1

[0088]

[0089] Data query interface services are mainly divided into real-time query and historical query:

[0090] Real-time query: Elasticsearch (real-time device location retrieval) + Redis (hot data cache);

[0091] Historical query: HBase SQL interface.

[0092] Application layer:

[0093] The application layer mainly implements the implementation of typical business application scenarios. Typical application modules include:

[0094] Safety monitoring: gas over-limit warning, underground personnel positioning and tracking;

[0095] Production optimization: ore sorting parameter adjustment, equipment predictive maintenance;

[0096] Management decisions: resource reserve analysis, production cost accounting reports.

[0097] The present invention focuses on the following aspects in optimizing the storage of mine streaming data and batch data:

[0098] 1. Data format and compression: For mine batch data, columnar format is used for storage, leveraging its efficient compression algorithm to improve data compression ratio and reduce storage space usage. For mine streaming data, during pre-processing at the mine edge computing node, it is aggregated by time window (such as 1 minute or 5 minutes) before storage, reducing data volume, storage pressure, and network transmission bandwidth requirements.

[0099] 2. Hot and cold data stratification: Based on the access frequency and time attributes of mine data, data is divided into hot data, warm data, and cold data. Hot data (such as underground environmental monitoring data from the last three days and real-time equipment operation data) is stored in high-performance storage devices, such as in-memory databases, to ensure fast access and meet the real-time monitoring requirements of mine production safety. Warm data (such as data within the last year) is stored in standard storage of the distributed file system. Cold data (such as data older than one year) is stored in archive storage to reduce storage costs.

[0100] Example 2:

[0101] An electronic device comprising a memory and a processor;

[0102] The memory is used to store computer programs;

[0103] The processor is configured to implement the method described in Example 1 when executing the computer program.

[0104] Example 3:

[0105] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described in Example 1 is implemented.

[0106] Example 4:

[0107] A computer program product includes a computer program, which implements the method described in embodiment 1 when executed by a processor.

[0108] In the above embodiments, references to "this embodiment" in the specification indicate that a particular feature, structure, or characteristic described in conjunction with the embodiment is included in at least some embodiments, but not necessarily all embodiments. Multiple occurrences of "this embodiment" do not necessarily refer to the same embodiment.

[0109] In the above embodiments, references to "this embodiment" in the specification indicate that a particular feature, structure, or characteristic described in conjunction with the embodiment is included in at least some embodiments, but not necessarily all embodiments. Multiple occurrences of "this embodiment" do not necessarily refer to the same embodiment.

[0110] In the above embodiments, although the invention has been described in conjunction with specific embodiments thereof, many alternatives, modifications, and variations of these embodiments will be apparent to those skilled in the art based on the foregoing description. For example, other memory structures (e.g., dynamic RAM (DRAM)) may be used with the embodiments discussed. The embodiments of the present invention are intended to encompass all such alternatives, modifications, and variations that fall within the broad scope of the appended claims.

[0111] Regarding the computer-readable storage medium in this embodiment, those skilled in the art will appreciate that all or part of the steps in the aforementioned method embodiments can be implemented using hardware associated with the computer program. The aforementioned computer program can be stored in a computer-readable storage medium. When executed, the program performs the steps in the aforementioned method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0112] The electronic terminal provided in this embodiment includes a processor, a memory, a transceiver and a communication interface. The memory and the communication interface are connected to the processor and the transceiver and complete communication with each other. The memory is used to store computer programs, the communication interface is used for communication, and the processor and the transceiver are used to run computer programs so that the electronic terminal executes the various steps of the above method.

[0113] In this embodiment, the memory may include a random access memory (RAM), and may also include a non-volatile memory (non-volatile memory), such as at least one disk storage.

[0114] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, and discrete hardware components.

[0115] The present invention can be used in a wide variety of general-purpose or special-purpose computing system environments or configurations, such as personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments that include any of the above.

[0116] The present invention may be described in the general context of computer-executable instructions, such as program modules, executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, and the like that perform specific tasks or implement specific abstract data types. The present invention may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communications network. In a distributed computing environment, program modules may be located in both local and remote computer storage media, including storage devices.

[0117] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not limiting. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solutions, which should all be included in the scope of the claims of the present invention.

Claims

1. A method for storing streaming data and massive batch data, characterized by: include: Classify mining data into streaming data and batch data; Build a hybrid storage architecture, including data collection layer, batch processing layer, speed layer, service layer, and application layer, so that streaming data and batch data are processed and stored using different rules. Specifically, Streaming data is collected using a collection adapter gateway and the real-time message queue Kafka; batch data is collected using the relational database DM8 and scheduled Airflow. The batch processing layer is used to divide data into cold data, hot data, and warm data, and store the data in a classified manner; The speed layer is used to process and store real-time streaming data; The service layer is used to integrate temporary stream processing data and batch processing result data, and provide data query interface services; The application layer is used to set up business application modules and access various databases to implement data applications.

2. The method for storing streaming data and massive batch data according to claim 1, wherein: The mining data is classified into streaming data and batch data, specifically including: A data acquisition and reception program is deployed on the mine side to receive real-time data from underground sensors and batch data submitted by the surface system. The data is classified using a recognition algorithm based on mine data features: If the data contains continuous timestamps and is transmitted in real-time stream form, it is determined to be coal mine streaming data; If the data is submitted in batches and the data volume is large, it is determined to be coal mine batch data.

3. The method for storing streaming data and massive batch data according to claim 1, wherein: The streaming data includes environmental monitoring, personnel and vehicle positioning, slope monitoring, tailings pond monitoring, and equipment operating condition data; the batch data includes report and ledger data; The environmental monitoring data includes methane, carbon monoxide, wind speed, temperature, and equipment start and stop, and is collected through databases, web APIs, and message queues. The personnel and vehicle positioning data includes personnel information, personnel real-time location, personnel trajectory, vehicle information and vehicle location, and is collected through databases and WEB APIs; The slope monitoring data includes slope displacement, stress, and groundwater; The tailings pond monitoring data includes tailings pond water level, rainfall, dry beach, infiltration line, displacement, and video; The equipment operating condition data includes the equipment's operating status, motor temperature, voltage, current, and power; The report ledger data includes production output or footage ledger, equipment maintenance and equipment failure records.

4. The method for storing streaming data and massive batch data according to claim 1, wherein: The batch processing layer divides the data into hot data, warm data and cold data according to the access frequency and time attributes of the mine data, uses HBase to store cold data, relational database DM8 to store warm data, and Redis memory database to store hot data.

5. The method for storing streaming data and massive batch data according to claim 1, wherein: The batch processing layer uses different partitioning rules for different data types, including: For equipment operating condition data, partition them according to the equipment's subsystem ID and data type; Partition production data by shift, mine or ore type; Spark Batch is used to process offline tasks, including monthly production statistics and equipment failure rate analysis.

6. The method for storing streaming data and massive batch data according to claim 1, wherein: The speed layer collects real-time streaming data from sensors, transmits it through Kafka message queues by topic partitions, processes it through Spark Streaming, and finally parses the streaming data by the real-time parsing module. If the real-time value exceeds the set alarm threshold, the alarm mechanism is triggered and the data is written to the alarm center. If the data is normal, the latest sampled real-time data is written to the Redis in-memory database. Aggregate device status based on a fixed time window and perform statistics on the data within the time window, including the maximum minute value, maximum time, minimum value, minimum time, average value, and highest level status of the monitoring point; For discrete quantities, calculate and count the state flip records of monitoring points.

7. The method for storing streaming data and massive batch data according to claim 1, wherein: Consider fault-tolerance strategies and use Kafka cluster partition replicas to cope with network disconnection or congestion.

8. The method for storing streaming data and massive batch data according to claim 1, wherein: Data integration at the service layer includes: For real-time device status, Redis stores the latest 5-minute data for temporary stream processing, and HBase stores the full historical data for batch processing results. The integration method prioritizes stream results for real-time queries and periodically uses batch results for corrections. For statistical data, the temporary data for stream processing is the hourly estimated value of stream computing, and the batch processing result data is the precise value at the end of the batch processing day. The integration method is to overwrite the stream results with the batch results at 24:00 every day.

9. The method for storing streaming data and massive batch data according to claim 1, wherein: The data query interface services of the service layer are divided into real-time query and historical query: Real-time queries include real-time device location retrieval in Elasticsearch and hot data cache in Redis; Historical queries use the HBase SQL interface.

10. The method for storing streaming data and massive batch data according to claim 1, wherein: The application layer includes the following applications: Safety monitoring: gas over-limit warning, underground personnel positioning and tracking; Production optimization: ore sorting parameter adjustment, equipment predictive maintenance; Management decisions: resource reserve analysis, production cost accounting reports.

Images