Coal mine data acquisition method and system based on script language

Abstract

The invention relates to the technical field of data acquisition, in particular to a coal mine data acquisition method and system based on a script language. Based on the wide data source access capability of the data source layer, the universal problem of multi-source heterogeneous data acquisition is solved, and unified access of various data sources such as databases, industrial equipment, files, Web and message queues is realized; by means of a standardized template of a script library layer and a multi-language execution environment of a script engine layer, customization and iteration of collection logic are achieved, the problem that a traditional method is insufficient in flexibility and expandability is solved, and protocol changes can be adapted without compiling and publishing; through a unified task management and resource isolation mechanism of a scheduling layer and data circulation and quality enhancement of a unified data bus, high reliability and data quality of the system are ensured, and stable and efficient management and control of large-scale acquisition tasks are realized; the method breaks through the rigid limitation of technology type selection, and improves the efficiency, flexibility and controllability of coal mine data collection.

Description

Technical Field

[0001] This invention relates to the field of data acquisition technology, and in particular to a coal mine data acquisition method and system based on a scripting language. Background Technology

[0002] The coal industry faces the challenge of acquiring multi-source heterogeneous data during its digital transformation. Existing data acquisition methods have three major drawbacks: First, they lack versatility, requiring the deployment of multiple independent tools for different data sources, resulting in complex systems and high maintenance costs. Second, they lack flexibility and scalability, requiring modification of source code, recompilation, and re-deployment of the entire system when the content or protocol changes, leading to severe architectural coupling. Third, they exhibit rigid technology selection, using a single development language to address all scenarios, resulting in bloated development for simple tasks while failing to meet the needs of specific scenarios. Summary of the Invention

[0003] Therefore, the technical problem to be solved by the present invention is to overcome the problems of poor universality, insufficient flexibility and scalability, and rigid technology selection in the existing coal mine data acquisition methods.

[0004] To address the aforementioned technical problems, this invention provides a coal mine data acquisition method based on a scripting language, comprising: S101: Based on the data source layer's ability to access multiple types of data sources, it collects raw data from different data sources through various communication protocols and connection methods, generating raw data streams for script processing; S102: Based on the standardized acquisition script templates preset by the script library layer, the corresponding acquisition scripts are called and executed through the script engine layer to extract, parse and transform the raw data stream and generate standardized intermediate data; S103: Based on the multi-language execution environment preset by the script engine layer, the collection script is run in a resource-isolated sandbox through multiple script engines to process standardized intermediate data and generate script output data; S104: Based on the unified scheduling and execution management capabilities of the scheduling layer, the task scheduler and resource management mechanism are used to schedule and control the collection tasks, and generate task execution status and logs. S105: A data channel based on a unified data bus, which improves the quality of script output data through data access, conversion, routing and output capabilities, and generates standardized data streams.

[0005] Preferably, based on the data source layer's ability to access multiple types of data sources, raw data is collected from different data sources through various communication protocols and connection methods to generate raw data streams for script processing, including: Based on a relational database source, full or incremental data collection is performed via JDBC or native connection protocols to generate structured raw data. Based on industrial equipment data sources, the system collects equipment status and process parameters in real time through industrial communication protocols to generate raw industrial equipment data. Based on the file system data source, the original file data is generated by formatting the file and performing file monitoring and incremental synchronization. Based on web data sources, we use RESTful APIs, SOAP, or web crawlers to call interfaces or extract page data to generate raw web data. Based on the message queue data source, consume data from the message middleware, process real-time data streams, and generate raw message data; Based on other extended data sources, and through the extended interfaces reserved in the architecture, a preset special data source can be accessed to generate extended raw data.

[0006] Preferably, based on the standardized data acquisition script templates preset in the script library layer, the corresponding acquisition scripts are called and executed through the script engine layer to extract, parse, and transform the raw data stream, generating standardized intermediate data including: Based on database acquisition scripts, structured raw data is processed and standardized structured data is generated by encapsulating connection management and data extraction logic. Based on the industrial protocol acquisition script, the raw data of industrial equipment is processed through protocol parsing and point mapping to generate standardized equipment data; Based on the FTP file acquisition script, the raw file data is processed through file monitoring, transmission and parsing to generate parsed file content data; Based on API web crawler scripts, this system manages HTTP requests and extracts data to process raw web data and generate structured web data. Based on the message queue collection script, the raw message data is processed through message subscription and deserialization to generate transformed message data; Based on other pre-set dedicated scripts, pre-set customized logic is used to process special data sources, expand the original data, and generate dedicated scenario data.

[0007] Preferably, based on the multi-language execution environment preset in the script engine layer, the collection script is run in a resource-isolated sandbox using multiple script engines to process standardized intermediate data and generate script output data including: Based on a Python script engine, the system integrates data processing libraries to execute logic and generate data processed by Python scripts. Based on the Lua script engine, industrial protocol parsing is performed through a lightweight virtual machine to generate data processed by Lua scripts; Based on a JavaScript engine, high-concurrency web scraping is performed through an event-driven model to generate data processed by JavaScript scripts; Based on extensions of other preset script engines, it integrates and executes preset language-specific scripts through plug-in integration to generate data processed by other preset script engines; Based on a resource isolation sandbox, memory, CPU, and network limits are used to ensure script execution safety and generate script output data.

[0008] Preferably, based on the unified scheduling and execution management capabilities of the scheduling layer, the data collection tasks are scheduled and controlled through a task scheduler and resource management mechanism, generating task execution status and logs, including: Based on the task scheduler, task execution instructions are generated through timed, event-driven, and manual triggering strategies. Based on resource management and isolation sandboxes, each task is run through Docker containers, generating isolated task instances. Based on task lifecycle management, task operation and maintenance data is generated through status monitoring, log collection, and anomaly alerts.

[0009] Preferably, based on a unified data bus data channel, through data access, conversion, routing, and output capabilities, the script output data is processed and its quality improved to generate a standardized data stream, including: Based on the data bus, a unified format data stream is generated through various serialization formats; Based on the data quality inspection module, the data is generated after quality inspection through integrity and accuracy verification and cleaning and transformation. Based on the data auditing function, traceable data metadata is generated by recording data lineage and operation logs.

[0010] Preferably, the coal mine data acquisition method based on scripting language further includes: Based on the storage, governance and service capabilities of the data management layer, standardized data streams are persisted and governed through multi-level storage and metadata management, generating data services that can be used by applications.

[0011] Preferably, based on the storage, governance, and service capabilities of the data management layer, standardized data streams are persisted and governed through multi-level storage and metadata management to generate data services available for application use, including: Based on real-time data storage, it provides low-latency data access and generates real-time data services through columnar databases and in-memory databases; Based on historical data storage, a data lake and tiered storage strategy are used to store all historical data and generate historical data services. Based on metadata management, data governance metadata is generated by recording data source, structure, and lineage information; Based on data standard management, standardized data is generated by executing naming, encoding, and format specifications. Based on data fusion services, a unified data view is built by associating, integrating, and enhancing multi-source data, generating fused data services.

[0012] Preferably, the coal mine data acquisition method based on scripting language further includes: Based on the operation and maintenance monitoring system and user interface, the system manages the system and tasks through web management, script debugging, and task configuration functions, and generates user operation feedback and system operation and maintenance data.

[0013] The present invention also provides a coal mine data acquisition device based on a scripting language, comprising: The data source layer is used to collect raw data from different data sources through various communication protocols and connection methods, and generate raw data streams for script processing; The script library layer is used to extract, parse, and transform the raw data stream and generate standardized intermediate data by calling and executing the corresponding collection scripts through the script engine layer based on the preset standardized collection script templates. The script engine layer is used to run collection scripts in a resource-isolated sandbox using various script engines according to a preset multi-language execution environment, process standardized intermediate data, and generate script output data. The scheduling layer is used to schedule and control the collection tasks through the task scheduler and resource management mechanism, and to generate task execution status and logs. A unified data bus is used to process and improve the quality of script output data based on data channels through data access, transformation, routing, and output capabilities, generating standardized data streams.

[0014] This invention also provides a coal mine data acquisition device based on a scripting language, comprising: Memory, used to store computer programs; A processor is used to implement the steps of the above-described coal mine data acquisition method based on a scripting language when executing the computer program.

[0015] The present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the above-described coal mine data acquisition method based on a scripting language.

[0016] The technical solution of the present invention has the following advantages compared with the prior art: The coal mine data acquisition method based on scripting language described in this invention achieves fundamental optimization of the data acquisition process through layered design and scripting technology, demonstrating significant beneficial effects. First, based on the broad data source access capabilities of the data source layer, it effectively solves the problem of universality in multi-source heterogeneous data acquisition, achieving unified access to various data sources such as databases, industrial equipment, files, Web, and message queues. Second, leveraging the standardized templates of the script library layer and the multi-language execution environment of the script engine layer, it enables flexible customization and rapid iteration of the acquisition logic, addressing the pain points of insufficient flexibility and scalability in traditional methods, and adapting to protocol changes without compilation and release. Furthermore, through the unified task management and resource isolation mechanism of the scheduling layer, and the data flow and quality enhancement of the unified data bus, it ensures high system reliability and data quality, achieving stable and efficient control over large-scale acquisition tasks. Finally, this method breaks through the rigid limitations of technology selection, significantly improving the efficiency, flexibility, and controllability of coal mine data acquisition. Attached Figure Description

[0017] To make the content of this invention easier to understand, the invention will be further described in detail below with reference to specific embodiments and accompanying drawings, wherein: Figure 1 This is a flowchart illustrating the implementation of a coal mine data acquisition method based on a scripting language provided by the present invention. Figure 2 This is an architecture diagram of a coal mine data acquisition method based on a scripting language provided in one embodiment of the present invention. Detailed Implementation

[0018] The core of this invention is to provide a coal mine data acquisition method and system based on a scripting language. Through scripting, multi-engine and hierarchical scheduling architecture, it effectively achieves universal adaptation, flexible expansion and stable operation of coal mine multi-source heterogeneous data acquisition, significantly reducing development complexity and operation and maintenance costs.

[0019] To enable those skilled in the art to better understand the present invention, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. Obviously, the described embodiments are merely some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] Please refer to Figure 1. Figure 1 The flowchart illustrates the implementation of a coal mine data acquisition method based on a scripting language provided by this invention; the specific operation steps are as follows: Based on the above embodiments, this embodiment will provide a detailed description of step S101: In some embodiments, based on the data source layer's ability to access multiple types of data sources, raw data is collected from different data sources through various communication protocols and connection methods to generate a raw data stream for script processing, including: • Based on a relational database source, full or incremental data collection is performed via JDBC or native connection protocols to generate structured raw data; It should be noted that the relational database source supports mainstream relational databases including MySQL, Oracle, SQL Server, PostgreSQL, etc., and extracts data through standard JDBC or native connection protocols. It supports two modes: full collection and incremental collection based on timestamps and auto-incrementing IDs.

[0021] • Based on industrial equipment data sources, and through industrial communication protocols, the system collects equipment status and process parameters in real time to generate raw industrial equipment data; It should be noted that the industrial equipment data source is designed for industrial IoT scenarios, supports various PLC controllers, sensor devices, and SCADA systems, and is compatible with industrial communication protocols such as Modbus TCP / RTU, OPC UA, OPC DA, and Profibus. It can collect equipment status, process parameters, quality data, etc. in real time.

[0022] • Based on the file system data source, it generates the original file data by formatting the file and performing file monitoring and incremental synchronization. It should be noted that the file system data source supports collecting file data from FTP servers, SFTP servers, local file systems, and network shared storage. It supports multiple formats such as text files, CSV, Excel, JSON, and XML, and has file monitoring, incremental synchronization, and breakpoint resume capabilities.

[0023] • Based on web data sources, use RESTful APIs, SOAP, or web crawlers to call interfaces or extract page data to generate raw web data; It should be noted that the Web data source supports data interface calls via RESTful API and SOAP Web service, and also has web crawling capabilities to extract structured data from website pages. It supports identity authentication, session management, and anti-crawler strategy countermeasures.

[0024] • Based on message queue data sources, consume data from message middleware, process real-time data streams, and generate raw message data; It should be noted that the message queue data source supports consuming data from message middleware such as Kafka, RabbitMQ, MQTT, and RocketMQ, can process real-time data streams, and supports multiple message serialization formats.

[0025] • Based on other extended data sources, through the extended interfaces reserved in the architecture, a preset special data source can be connected to generate extended raw data.

[0026] It should be noted that other extended data sources: the architecture reserves extension interfaces, which can be used to connect to special data sources such as NoSQL databases, IoT platforms, and cloud service APIs according to business needs.

[0027] Based on the above embodiments, this embodiment will provide a detailed description of step S102: In some embodiments, based on the standardized acquisition script templates preset in the script library layer, the corresponding acquisition scripts are called and executed through the script engine layer to extract, parse, and transform the raw data stream, generating standardized intermediate data, including: • Based on database acquisition scripts, structured raw data is processed and standardized structured data is generated by encapsulating connection management and data extraction logic; It should be noted that the database collection script encapsulates the connection management and data extraction logic of various databases, supports incremental collection strategies based on timestamps, auto-incrementing IDs, and data version numbers, and provides general functions such as data pagination, batch extraction, and connection pool management.

[0028] • Based on industrial protocol acquisition scripts, raw data from industrial equipment is processed through protocol parsing and point mapping to generate standardized equipment data; It should be noted that the industrial protocol acquisition script: realizes the parsing and adaptation of industrial communication protocols, converts the proprietary protocols of different manufacturers' equipment into standard data formats, and supports industrial-specific functions such as device register mapping, data point configuration, and acquisition frequency control.

[0029] • Based on FTP file acquisition scripts, the scripts monitor, transmit, and parse files to process raw file data and generate parsed file content data. It should be noted that the FTP file acquisition script provides a complete solution for file monitoring, file transfer, and file parsing, and supports file processing functions such as automatic file encoding detection, compressed file decompression, file content verification, and duplicate file filtering.

[0030] • Based on API web crawler scripts, it processes raw web data and generates structured web data through HTTP request management and data extraction; It should be noted that the API web crawler script encapsulates common logic such as HTTP request management, authentication, and data parsing. It supports API rate limiting, request retry mechanisms, and anti-crawler bypass strategies, and provides multiple data extraction methods such as XPath, CSS selectors, and regular expressions.

[0031] • Based on the message queue collection script, the raw message data is processed through message subscription and deserialization to generate transformed message data; It should be noted that the message queue collection script implements a standard process for message subscription, deserialization, and data conversion, supports parsing of multiple message formats, and provides message processing capabilities such as consumer position management, message replay, and consumer group coordination.

[0032] • Based on other preset dedicated scripts, it processes preset special data sources through preset customized logic, processes extended original data, and generates dedicated scene data.

[0033] It should be noted that other preset scripts are dedicated data collection scripts developed for specific business scenarios, such as log file collection, video stream collection, sensor data collection, and other special needs.

[0034] Based on the above embodiments, this embodiment will provide a detailed description of step S103: In some embodiments, based on a pre-defined multi-language execution environment at the script engine layer, the acquisition script is run in a resource-isolated sandbox using multiple script engines to process standardized intermediate data and generate script output data, including: • Based on a Python scripting engine, it generates data processed by Python scripts by integrating data processing libraries to execute logic. It should be noted that the Python script engine, based on the CPython runtime, provides complete Python language support, integrates commonly used libraries for data processing, network communication, and file operations, supports virtual environment isolation for third-party dependencies, and is suitable for complex data processing scenarios.

[0035] • Based on the Lua scripting engine, industrial protocol parsing is performed through a lightweight virtual machine to generate data processed by Lua scripts; It should be noted that the Lua scripting engine uses a lightweight Lua virtual machine, which starts up quickly and consumes few resources, making it particularly suitable for embedded environments and performance-sensitive scenarios. It has advantages in industrial protocol parsing and network data processing.

[0036] • Based on a JavaScript engine, it performs high-concurrency web scraping through an event-driven model, generating data processed by JavaScript scripts; It should be noted that the JavaScript engine is based on the Node.js runtime, supports modern JavaScript features, and utilizes an event-driven, non-blocking I / O model, making it suitable for high-concurrency, I / O-intensive web data acquisition tasks.

[0037] • Based on extensions of other preset script engines, it integrates and executes preset language-specific scripts through plug-in integration to generate data processed by other preset script engines; It should be noted that other script engine extensions: the architecture design supports plug-in integration of other language engines, such as the Go language engine suitable for high-performance concurrent data collection, the Java engine suitable for integration with existing enterprise Java systems, and the Shell engine suitable for system command call integration.

[0038] • Based on a resource isolation sandbox, memory, CPU, and network limits are used to ensure script execution safety and generate script output data.

[0039] It should be noted that all script engines run in a resource-isolated sandbox environment, providing security mechanisms such as memory limits, CPU quotas, network access control, and file system isolation to prevent malicious scripts from damaging the system.

[0040] Based on the above embodiments, this embodiment will provide a detailed description of step S104: In some embodiments, based on the unified scheduling and execution management capabilities of the scheduling layer, the task scheduler and resource management mechanism are used to schedule and control the data collection tasks, generating task execution status and logs, including: • Based on the task scheduler, task execution instructions are generated through timed, event-driven, and manual triggering strategies; It should be noted that the task scheduler, based on a distributed scheduling architecture, supports concurrent scheduling of tens of thousands of tasks and provides various triggering strategies, including timed triggering, event triggering, manual triggering, and dependency triggering. It also supports complex scheduling strategy configurations, such as advanced features like task priority, load balancing, failover, and holiday scheduling.

[0041] • Based on resource management and isolation sandboxes, each task is run through Docker containers, generating isolated task instances. It should be noted that resource management and isolation sandboxing: Containerization technology is used to achieve task-level resource isolation. Each data collection task runs in an independent Docker container, ensuring environment and security isolation. Fine-grained resource quota management is provided, including CPU core limits, memory usage limits, disk space quotas, and network bandwidth control.

[0042] • Based on task lifecycle management, task operation and maintenance data is generated through status monitoring, log collection, and anomaly alerts.

[0043] It should be noted that the scheduling layer is also responsible for the lifecycle management of tasks, including task status monitoring, execution log collection, performance metric collection, and anomaly alarm notification and other operation and maintenance functions.

[0044] Based on the above embodiments, this embodiment will provide a detailed description of step S105: In some embodiments, based on a unified data bus data channel, the script output data is processed and its quality improved through data access, transformation, routing, and output capabilities to generate a standardized data stream, including: • Based on the data bus, a unified format data stream is generated through various serialization formats; It should be noted that the data bus adopts the Enterprise Service Bus design pattern, providing unified data access, transformation, routing, and output capabilities. It supports multiple data serialization formats, including JSON, Avro, and Protobuf, ensuring the efficiency and quality of data transmission across systems.

[0045] • Based on the data quality inspection module, data is generated after quality inspection through integrity and accuracy verification and cleaning and transformation; It should be noted that the bus has a built-in data quality inspection module that verifies the integrity, accuracy, and consistency of the collected data, and supports data preprocessing operations such as data cleaning, format conversion, and encoding.

[0046] • Based on the data auditing function, traceable data metadata is generated by recording data lineage and operation logs.

[0047] It should be noted that the bus also provides data auditing functionality, recording complete data lineage information to meet data governance and compliance requirements.

[0048] like Figure 2 Based on the above embodiments, the coal mine data acquisition method based on scripting language further includes: Based on the storage, governance and service capabilities of the data management layer, standardized data streams are persisted and governed through multi-level storage and metadata management, generating data services that can be used by applications.

[0049] In some embodiments, based on the storage, governance, and service capabilities of the data management layer, standardized data streams are persisted and governed through multi-level storage and metadata management to generate data services available for application use, including: • Based on real-time data storage, it provides low-latency data access and generates real-time data services through columnar databases and in-memory databases; It should be noted that real-time data storage employs columnar storage database and in-memory database technologies to provide low-latency data access for scenarios such as real-time monitoring, instant querying, and business dashboards. It stores recently accessed data, typically retaining it for 30 to 90 days, and supports high-concurrency read and write operations.

[0050] • Based on historical data storage, a data lake and tiered storage strategy are used to store all historical data and generate historical data services; It should be noted that historical data storage: An enterprise data lake is built based on distributed file systems and object storage technologies to store all historical data. A tiered storage strategy is adopted, automatically migrating data between different storage media based on data access frequency, balancing storage costs and access performance.

[0051] • Based on metadata management, data governance metadata is generated by recording data source, structure, and lineage information; It should be noted that metadata management involves establishing a complete metadata management system, including technical metadata, business metadata, and operational metadata. This records metadata such as data source connection information, data structure, data lineage, data quality metrics, and access permissions, providing support for data discovery, data understanding, and data governance.

[0052] • Based on data standards management, standardized data is generated by executing naming, encoding, and format specifications; It should be noted that data standards management involves developing and implementing enterprise data standards, including data naming conventions, coding rules, data formats, accuracy requirements, and unit standards, to ensure the consistency and accuracy of data throughout its entire lifecycle.

[0053] • Based on data fusion services, a unified data view is built by associating, integrating, and enhancing multi-source data, generating fused data services.

[0054] It should be noted that the data fusion service provides data fusion capabilities such as data association, data integration, and data enhancement, which can perform correlation analysis on heterogeneous data from different data sources, build a unified data view, and support cross-domain data analysis and business applications.

[0055] Based on the above embodiments, the coal mine data acquisition method based on scripting language further includes: Based on the operation and maintenance monitoring system and user interface, the system manages the system and tasks through web management, script debugging, and task configuration functions, and generates user operation feedback and system operation and maintenance data.

[0056] In some embodiments, based on the operation and maintenance monitoring system and user interface, the system and tasks are managed through web management, script debugging, and task configuration functions, generating user operation feedback and system operation and maintenance data, including: • Based on the operation and maintenance monitoring system, system operation and maintenance indicators are generated by monitoring the system health status, resource usage, and task execution status in real time. • Based on the intelligent alarm mechanism, system anomalies are promptly notified through multiple channels, and alarm information is generated; • Based on the web management interface, users can complete data source configuration, data collection task definition, scheduling strategy setting, and generate user operation feedback through a graphical interface. • Based on script development tools, it uses online script development, debugging, and testing tools, as well as built-in script templates and data collection test cases, to process user operation feedback and generate deployable script versions.

[0057] This invention provides a coal mine data acquisition system based on a scripting language; specifically, it may include: The data source layer is used to collect raw data from different data sources through various communication protocols and connection methods, and generate raw data streams for script processing; It should be noted that the data source layer is the system's input endpoint, supporting data collection from various types of data sources.

[0058] The script library layer is used to extract, parse, and transform the raw data stream and generate standardized intermediate data by calling and executing the corresponding collection scripts through the script engine layer based on the preset standardized collection script templates. It should be noted that the script library layer provides standardized data collection script templates, enabling the reuse and standardized management of data collection logic; The script engine layer is used to run collection scripts in a resource-isolated sandbox using various script engines according to a preset multi-language execution environment, process standardized intermediate data, and generate script output data. It should be noted that the script engine layer provides a multi-language execution environment to ensure the safe and efficient execution of scripts; The scheduling layer is used to schedule and control the collection tasks through the task scheduler and resource management mechanism, and to generate task execution status and logs. It should be noted that the scheduling layer is responsible for the unified scheduling and execution management of data collection tasks; A unified data bus is used to process and improve the quality of script output data based on data channels through data access, transformation, routing, and output capabilities, generating standardized data streams.

[0059] It should be noted that the unified data bus is the core channel for data flow within the system.

[0060] The coal mine data acquisition system also includes: The data management layer is used to persist and govern standardized data streams through multi-level storage and metadata management, generating data services that can be used by applications. It should be noted that the data management layer is responsible for the storage, governance, and service of the collected data.

[0061] Based on the above embodiments, the core features of the system are as follows: High scalability: The system adopts a microkernel architecture, maintaining a stable core framework, and each functional module supports hot-swapping. New data source types can be quickly integrated by developing corresponding acquisition scripts, and new scripting language engines can be integrated into the system by implementing standard interfaces. Both the scheduler and executor support horizontal scaling, enabling cluster deployment to meet ever-increasing data acquisition demands.

[0062] High reliability: The system possesses a robust fault tolerance mechanism and fault recovery capability. Automatic retries occur when tasks fail, and custom retry strategies and backoff algorithms are supported. Critical data acquisition supports breakpoint resumption, ensuring the integrity of large-scale data transmission. The system provides multi-level backup strategies, with persistent storage for core configurations and data to prevent data loss due to single points of failure.

[0063] Enterprise-level security: A comprehensive security protection system is established, including three layers of security controls: identity authentication, access authorization, and operation auditing. Data source credentials are stored encrypted, and TLS encryption is used during transmission. The script execution environment is strictly isolated to prevent unauthorized access. Complete user operation logs and data access logs are recorded to meet security auditing and compliance requirements.

[0064] Operational friendliness: Provides a complete operation and maintenance monitoring system, monitoring system health status, resource usage, and task execution status in real time. Establishes an intelligent alarm mechanism to promptly notify of system anomalies through multiple channels. Offers a wealth of operation and maintenance tools, including task configuration, script management, system monitoring, and log querying, reducing the complexity of system operation and maintenance.

[0065] User-friendliness: Provides an intuitive web management interface, allowing users to configure data sources, define collection tasks, and set scheduling strategies through a graphical interface. Includes a rich set of built-in script templates and collection test cases, lowering the barrier to entry for users. Offers online script development, debugging, and testing tools to improve script development efficiency.

[0066] Based on the above embodiments, the core of this system lies in decoupling, abstracting, and standardizing the complex process of data acquisition through scripting technology. This not only changes the development model but also fundamentally improves the efficiency, flexibility, and controllability of data acquisition. The specific technical effects are reflected in: (1) Extremely flexible, enabling "everything can be harvested". Results: By leveraging the programming capabilities of scripts, the protocol and format compatibility issues that are unavoidable with any standardized data collection tool are overcome.

[0067] Specifically, for non-standard or proprietary protocols (such as vendor-specific device protocols or custom TCP / UDP packets), they can be easily parsed by writing custom scripts, without waiting for official connectors from vendors or the community. For data sources with complex structures or dynamic changes (such as web pages requiring complex login authentication or APIs returning nested JSON), scripts can flexibly handle interaction logic and data structure parsing, which is difficult to achieve with configuration tools. They can implement complex business logic, such as data cleaning, format conversion, field anonymization, and multi-table joins during the collection process, achieving "collection as processing" and reducing the burden on downstream systems.

[0068] (2) Efficient development, from “coding” to “assembly” Effect: The development mode of data collection tasks has been changed from "project-based" hard coding to "component-based" script assembly, which greatly improves development efficiency.

[0069] Specifically, this manifests in several ways: Script template library and reuse: Pre-built, validated script templates are available for common data collection scenarios (such as incremental database synchronization and FTP file monitoring). Developers don't need to start from scratch; they can quickly create a stable and reliable data collection task by modifying key configuration parameters (such as connection addresses and table names), much like "assembling parts." Focus on business logic, shielding from underlying complexity: Script developers don't need to worry about the complexities of the underlying platform, such as task scheduling, distributed execution, fault recovery, and resource isolation. They can focus on the core data extraction and parsing logic, significantly reducing development difficulty and code volume.

[0070] (3) Lower the barrier to entry Results: It breaks the dependence of data collection on a specific technology stack, allowing engineers with different technical backgrounds to participate.

[0071] Specifically, multi-language engine support means that data analysts skilled in data processing can easily write API and database collection scripts using Python (Pandas, Requests). Industrial engineers familiar with embedded systems and network protocols can write device communication scripts using lightweight Lua. Front-end or full-stack engineers can handle web crawling and JSON API collection using JavaScript. Enterprises do not need to enforce a unified technology stack and can fully leverage the technical diversity of their existing teams to maximize the value of their human resources.

[0072] (4) Stable and reliable, the script runs without any problems. Effect: Provides a robust and reliable "runtime sandbox" for flexible but potentially fragile scripts, ensuring their stable execution.

[0073] Specifically, this manifests in: Resource isolation and security sandboxing: Each script runs in an independent container. Even if a script gets stuck in an infinite loop or memory leak due to coding issues, it will be confined within its own sandbox, preventing it from dragging down the entire data collection platform or other tasks, thus ensuring the overall stability of the system. Comprehensive platform-level safeguards: The scripts themselves do not need to handle "operational-level" issues such as retries, fault tolerance, and monitoring. The platform provides functions such as automatic retries, breakpoint resumption, comprehensive monitoring, and alerts. For example, if a database data collection script is interrupted due to network jitter, the platform will automatically retry and resume from the breakpoint after recovery, without the script logic being aware of this.

[0074] Unified credential management: There is no need to hardcode sensitive information such as usernames and passwords in the script. They are obtained securely through the platform's unified credential management service, which is both secure and easy to rotate.

[0075] (5) Convenient operation and maintenance and end-to-end observability Results: Enabled unified and standardized management of tens of thousands of data collection scripts.

[0076] Specifically, this manifests in two ways: Centralized Management: All scripts, their versions, configurations, and dependencies are stored in a unified script repository, providing a clear overview and avoiding the management chaos caused by scripts being scattered across personal computers. Standardized Operations and Maintenance: Through a unified web interface, all scripted data collection tasks can be started, stopped, scheduled, configured, and their performance monitored, making operations and maintenance simple and consistent.

[0077] Complete observability: The platform fully records and displays the execution logs, output data, running status, and resource consumption of each script. When data issues arise, it can quickly pinpoint whether the problem lies with the data source or the script logic, achieving end-to-end data lineage tracing from the data source to the target.

[0078] The scripting language-based coal mine data acquisition system of this embodiment is used to implement the aforementioned scripting language-based coal mine data acquisition method. Therefore, the specific implementation of the scripting language-based coal mine data acquisition system can be found in the previous embodiment section of the scripting language-based coal mine data acquisition method. For example, the data source layer, script library layer, script engine layer, scheduling layer, and unified data bus are used to implement steps S101, S102, S103, S104, and S105 in the above-mentioned scripting language-based coal mine data acquisition method. Therefore, its specific implementation can be referred to the description of the corresponding embodiments, which will not be repeated here.

[0079] Obviously, the above embodiments are merely illustrative examples for clear explanation and are not intended to limit the implementation. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is neither necessary nor possible to exhaustively list all possible implementations here. However, obvious variations or modifications derived therefrom are still within the scope of protection of this invention.

Claims

1. A coal mine data acquisition method based on a scripting language, characterized in that, include: Based on the data source layer's ability to access various types of data sources, raw data is collected from different data sources through various communication protocols and connection methods to generate raw data streams for script processing; Based on the standardized data acquisition script templates preset by the script library layer, the corresponding data acquisition scripts are called and executed through the script engine layer to extract, parse and transform the raw data stream and generate standardized intermediate data. Based on the multi-language execution environment preset by the script engine layer, the collection script is run in a resource-isolated sandbox through multiple script engines to process standardized intermediate data and generate script output data; Based on the unified scheduling and execution management capabilities of the scheduling layer, the task scheduler and resource management mechanism are used to schedule and control the collection tasks, and generate task execution status and logs. Based on a unified data bus, the data channel enables data access, transformation, routing, and output capabilities to streamline and improve the quality of script output data, generating standardized data streams.

2. The coal mine data acquisition method based on scripting language according to claim 1, characterized in that, The aforementioned capability of accessing multiple types of data sources based on the data source layer, through various communication protocols and connection methods, collects raw data from different data sources and generates raw data streams for script processing, including: Based on a relational database source, full or incremental data collection is performed via JDBC or native connection protocols to generate structured raw data. Based on industrial equipment data sources, the system collects equipment status and process parameters in real time through industrial communication protocols to generate raw industrial equipment data. Based on the file system data source, the original file data is generated by formatting the file and performing file monitoring and incremental synchronization. Based on web data sources, we use RESTful APIs, SOAP, or web crawlers to call interfaces or extract page data to generate raw web data. Based on the message queue data source, consume data from the message middleware, process real-time data streams, and generate raw message data; Based on other extended data sources, and through the extended interfaces reserved in the architecture, a preset special data source can be connected to generate extended raw data.

3. The coal mine data acquisition method based on scripting language according to claim 2, characterized in that, The standardized data acquisition script templates pre-set in the script library layer are called and executed by the script engine layer to extract, parse, and transform the raw data stream, generating standardized intermediate data, including: Based on database acquisition scripts, structured raw data is processed and standardized structured data is generated by encapsulating connection management and data extraction logic. Based on the industrial protocol acquisition script, the raw data of industrial equipment is processed through protocol parsing and point mapping to generate standardized equipment data; Based on the FTP file acquisition script, the raw file data is processed through file monitoring, transmission and parsing to generate parsed file content data; Based on API web crawler scripts, this system manages HTTP requests and extracts data to process raw web data and generate structured web data. Based on the message queue collection script, the raw message data is processed through message subscription and deserialization to generate transformed message data; Based on other pre-set dedicated scripts, pre-set customized logic is used to process special data sources, expand the original data, and generate dedicated scenario data.

4. The coal mine data acquisition method based on scripting language according to claim 1, characterized in that, The pre-defined multi-language execution environment based on the script engine layer runs the collection scripts in a resource-isolated sandbox using multiple script engines, processes standardized intermediate data, and generates script output data including: Based on a Python script engine, the system integrates data processing libraries to execute logic and generate data processed by Python scripts. Based on the Lua script engine, industrial protocol parsing is performed through a lightweight virtual machine to generate data processed by Lua scripts; Based on a JavaScript engine, high-concurrency web scraping is performed through an event-driven model to generate data processed by JavaScript scripts; Based on extensions of other preset script engines, it integrates and executes preset language-specific scripts through plug-in integration to generate data processed by other preset script engines; Based on a resource isolation sandbox, memory, CPU, and network restrictions are used to ensure script execution safety and generate script output data.

5. The coal mine data acquisition method based on scripting language according to claim 1, characterized in that, The unified scheduling and execution management capability based on the scheduling layer, through a task scheduler and resource management mechanism, schedules and controls the data collection tasks, and generates task execution status and logs, including: Based on the task scheduler, task execution instructions are generated through timed, event-driven, and manual triggering strategies. Based on resource management and isolation sandboxes, each task is run through Docker containers, generating isolated task instances. Based on task lifecycle management, task operation and maintenance data is generated through status monitoring, log collection, and anomaly alerts.

6. The coal mine data acquisition method based on scripting language according to claim 1, characterized in that, The data channel based on the unified data bus, through its data access, conversion, routing, and output capabilities, processes and improves the quality of script output data to generate standardized data streams, including: Based on the data bus, a unified format data stream is generated through various serialization formats; Based on the data quality inspection module, the data is generated after quality inspection through integrity and accuracy verification and cleaning and transformation. Based on the data auditing function, traceable data metadata is generated by recording data lineage and operation logs.

7. The coal mine data acquisition method based on scripting language according to claim 1, characterized in that, Also includes: Based on the storage, governance and service capabilities of the data management layer, standardized data streams are persisted and governed through multi-level storage and metadata management, generating data services that can be used by applications.

8. The coal mine data acquisition method based on scripting language according to claim 7, characterized in that, The storage, governance, and service capabilities based on the data management layer, through multi-level storage and metadata management, persist and govern standardized data streams to generate data services available for application use, including: Based on real-time data storage, it provides low-latency data access and generates real-time data services through columnar databases and in-memory databases; Based on historical data storage, a data lake and tiered storage strategy are used to store all historical data and generate historical data services. Based on metadata management, data governance metadata is generated by recording data source, structure, and lineage information; Based on data standard management, standardized data is generated by executing naming, encoding, and format specifications. Based on data fusion services, a unified data view is built by associating, integrating, and enhancing multi-source data, generating fused data services.

9. The coal mine data acquisition method based on scripting language according to claim 1, characterized in that, Also includes: Based on the operation and maintenance monitoring system and user interface, the system manages the system and tasks through web management, script debugging, and task configuration functions, and generates user operation feedback and system operation and maintenance data.

10. A coal mine data acquisition system based on a scripting language, characterized in that, include: The data source layer is used to collect raw data from different data sources through various communication protocols and connection methods, and generate raw data streams for script processing; The script library layer is used to extract, parse, and transform the raw data stream and generate standardized intermediate data by calling and executing the corresponding collection scripts through the script engine layer based on the preset standardized collection script templates. The script engine layer is used to run collection scripts in a resource-isolated sandbox using various script engines according to a preset multi-language execution environment, process standardized intermediate data, and generate script output data. The scheduling layer is used to schedule and control the collection tasks through the task scheduler and resource management mechanism, and to generate task execution status and logs. A unified data bus is used to process and improve the quality of script output data based on data channels through data access, transformation, routing, and output capabilities, generating standardized data streams.

Images