Metadata-driven two-dimensional standard data automatic storage method

Abstract

The present application relates to a kind of metadata-driven two-dimensional standard data automatic storage method, belong to coal mine informatization technical field, including the following steps: S1: construction metadata system and define two-dimensional standard data model specification, generate standardization data model specification document, the metadata system includes data structure metadata, constraint metadata and storage adaptation metadata, the model specification is clear two-dimensional data's row dimension rule and automatic storage trigger condition;S2: by ETL tool, establish target data with the dynamic mapping association of the data model specification, generate and include field mapping relationship, conversion rule and storage routing information's mapping configuration;S3: based on the data model specification and mapping configuration, trigger automatic storage process, including automatically parsing model specification generation storage structure, automatically match storage medium, automatically execute data write and check, realize the automatic storage of two-dimensional data.

Description

Technical Field

[0001] This invention belongs to the field of coal mine information technology and relates to a metadata-driven method for automatic storage of two-dimensional standard data. Background Technology

[0002] In the field of coal mine data management, the storage of two-dimensional relational exchange data (such as tabular data exchange data, text data exchange data, and API-type JSON data) in mine monitoring systems has long relied on manual intervention, resulting in the following problems: 1. Fragmented storage process: It requires manual definition of table structure, writing of storage scripts, and execution of data writing, which is cumbersome and prone to errors; 2. Disconnect between model and storage: After changes in the data model, the storage structure and related writing programs need to be manually adjusted, making it difficult to ensure consistency; 3. Poor storage adaptability: Different types and structures of two-dimensional data require manual writing of storage methods and mechanisms, lacking an automated adaptation mechanism; 4. Passive triggering mechanism: The storage process needs to be started manually and cannot be automatically triggered based on data volume, time, or other conditions, affecting timeliness.

[0003] In existing technologies, metadata is mostly used to describe data attributes and is not deeply integrated into the storage process; although ETL tools support data transformation, the storage process still relies on manual configuration. Therefore, there is an urgent need for a technical solution that uses metadata as the core driving force to achieve a closed loop of "model definition - association mapping - automatic storage" for two-dimensional relational exchange data in mine monitoring systems. Summary of the Invention

[0004] In view of this, the purpose of this invention is to provide a metadata-driven automatic storage method for two-dimensional standard data, which realizes full-process automation of two-dimensional exchange relational data in mine monitoring systems from model definition to final storage, and solves problems such as fragmented storage process, disconnect between model and storage, poor storage adaptability and passive triggering mechanism in the prior art, thereby improving storage efficiency and data consistency.

[0005] To achieve the above objectives, the present invention provides the following technical solution:

[0006] A metadata-driven method for automatically storing two-dimensional standard data includes the following steps: S1: Construct a metadata system and define a two-dimensional standard data model specification, and generate a standardized data model specification document. The metadata system includes data structure metadata, constraint metadata and storage adaptation metadata. The model specification clarifies the row and column dimension rules of two-dimensional data and the automatic storage triggering conditions. S2: Establish a dynamic mapping relationship between the target data and the data model specification through ETL tools, and generate a mapping configuration that includes field mapping relationships, transformation rules and storage routing information; S3: Based on the data model specification and mapping configuration, trigger the automatic storage process, including automatically parsing the model specification to generate the storage structure, automatically matching the storage medium, and automatically performing data writing and verification to realize the automatic storage of two-dimensional data.

[0007] Furthermore, in step S1, the data structure metadata definition includes entity name, row dimension identifier, column dimension attribute and row-column association rules; the constraint metadata definition includes data validation rules and storage constraint rules; the storage adaptation metadata definition includes storage medium type, storage engine parameters and automatic storage trigger conditions; and a standardized model specification document containing a metadata dictionary, a two-dimensional data model ER diagram and an automatic storage rule list is generated.

[0008] Furthermore, in step S1, the automatic storage triggering conditions include three methods: time-triggered, data volume-triggered, and event-triggered. Among them, time-triggered storage is automatically triggered at a preset time interval, data volume-triggered storage is triggered when the data volume reaches a preset threshold, and event-triggered storage is triggered when a data source change event is detected.

[0009] Furthermore, in step S2, establishing a dynamic mapping relationship between the target data and the data model specification using an ETL tool includes: Data source adaptation: Use ETL tools to automatically identify and adapt the target data source type and configure connection parameters; Intelligent field mapping: Automatically generates initial field mappings based on field name similarity and data type consistency, and supports manual intervention and adjustment; Conversion rule configuration: Automatically generate conversion scripts such as type conversion, format standardization, and unit conversion based on model specifications and write them into the mapping configuration; For data without field names, field matching is performed manually, and the field mapping is written into the mapping configuration.

[0010] Furthermore, in step S2, the conversion rules include type conversion, format standardization, and unit conversion rules; the storage routing information includes the target storage medium address, storage object identifier, and partition / shard key.

[0011] Furthermore, in step S3, the automatic storage process includes: S31: Automatic storage structure generation: Parses the data model specification and automatically generates the physical structure of the target storage medium; in relational database scenarios, it automatically generates table creation SQL; when a change in the model specification is detected, it automatically generates a structure change script and updates the storage medium synchronously. S32: Automatic storage media matching: Automatically detects the target storage environment based on storage adaptation metadata, automatically switches to backup storage and records the switchover log when the primary storage fails; manages connection pool parameters to ensure write performance; S33: Automatic data writing: Performs data extraction, transformation and writing according to mapping configuration and trigger conditions, supports batch writing and real-time writing, and locates the writing target according to partition / table rules; S34: Automatic verification and error correction of storage results: After writing is completed, structure verification, data verification and integrity verification are automatically performed; structure verification is used to verify the consistency between the storage structure and the model specification; data verification verifies the consistency between the written data and the source data through sampling comparison; integrity verification verifies the uniqueness of the primary key and the validity of the foreign key association; when the verification fails or abnormal data occurs, the exception is automatically written to the error log and the retry mechanism is triggered; if the retry fails, an alarm notification is issued.

[0012] Furthermore, in step S31, when a change in the model specification is detected, a change script is generated and verified in the test / pre-release environment first. After the verification is passed, the structure update is automatically executed sequentially in the production environment.

[0013] The beneficial effects of this invention are as follows: By constructing a model specification centered on metadata and integrating metadata throughout the ETL mapping and storage execution process, this invention achieves closed-loop automation of the "model definition—mapping configuration—automatic storage—verification and error correction" process for two-dimensional exchange relational data in mine monitoring systems. Compared with existing technologies, the advantages of this invention include, but are not limited to: I. Significantly reduces the need for manual intervention: Metadata-driven throughout the process, automatically generating storage structures and transformation rules, greatly reducing the workload of manually creating tables, writing storage scripts, and configuring operations; II. Improve consistency and reliability: The physical structure is automatically synchronized after model changes, and the closed-loop verification and retry mechanism ensures the accuracy and integrity of the written data; 3. Enhanced adaptability and timeliness: Supports multiple triggering strategies (time / data volume / event) and automatic matching with multiple media to achieve "data ready for storage"; IV. Easy to operate and expand: Automatic storage switching, connection pool management and partitioning strategies support large-scale data writing and smooth expansion.

[0014] Other advantages, objectives, and features of the invention will be set forth in part in the description which follows, and in part will be apparent to those skilled in the art from the following examination, or may be learned from practice of the invention. The objectives and other advantages of the invention can be realized and obtained through the following description. Attached Figure Description

[0015] To make the objectives, technical solutions, and advantages of the present invention clearer, the preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings, wherein: Figure 1This is a flowchart of a gas over-limit early warning method based on Bollinger Band change characteristics of gas monitoring data. Detailed Implementation

[0016] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0017] It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Therefore, the drawings only show the components related to the present invention and are not drawn according to the actual number, shape and size of the components in the actual implementation. In the actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0018] In the following description, numerous details are explored to provide a more thorough explanation of embodiments of the invention. However, it will be apparent to those skilled in the art that embodiments of the invention may be practiced without these specific details. In other embodiments, well-known structures and devices are shown in block diagram form rather than in detail to avoid obscuring embodiments of the invention.

[0019] Example 1: like Figure 1 As shown in the diagram, this embodiment provides a flowchart of a metadata-driven automatic storage method for two-dimensional standard data, which specifically includes the following steps: S1: Construct a metadata system and define a two-dimensional standard data model specification.

[0020] The metadata system includes data structure metadata, constraint metadata, and storage adaptation metadata. Data structure metadata defines entity names, row dimension identifiers (unique keys for entity instances), column dimension attributes (field names, data types, lengths, precision), and row-column association rules (e.g., "row key + column identifier → unique data unit"). Constraint metadata defines data validation rules (format, value range, uniqueness) and storage constraint rules (e.g., auto-incrementing primary keys, automatic matching of partition fields to time granularity). Storage adaptation metadata defines storage medium types (relational / non-relational databases), storage engine parameters (e.g., "MySQL's InnoDB, MongoDB's sharding key"), and automatic storage trigger conditions. Automatic storage trigger conditions include three methods: time-triggered, data volume-triggered, and event-triggered. Time-triggered storage is triggered automatically at preset time intervals; data volume-triggered storage is triggered when the data volume reaches a preset threshold; and event-triggered storage is triggered when a data source change event is detected, such as batch storage triggered when the data volume reaches 1000 records, and real-time storage triggered every minute for incremental data.

[0021] This step also generates a standardized data model specification document, which includes a metadata dictionary, a two-dimensional data model ER diagram, and a list of automatic storage rules.

[0022] S2: Establish a dynamic mapping relationship between the target data and the data model specification using ETL tools.

[0023] ETL tools read metadata specifications and automatically identify the target data source type (relational database, file, API interface, etc.), and automatically configure connection parameters (connection string, authentication information, pagination parameters, etc.) based on metadata adaptation information. Intelligent field mapping: Automatically generates initial field mappings based on field name similarity and data type consistency (e.g., mapping the target field "device_id" to "device code" in the model), and provides a manual intervention interface to adjust the mapping relationship; for two-dimensional relational data without field names (e.g., files using row and column placeholders or special delimiters), field parsing rules and manual adjustment tools are provided. Automatic configuration of conversion rules: Automatically generate conversion scripts (such as string to date, unit conversion, percentage text to number) based on the field type and format requirements of the model, and write the conversion rules into the mapping configuration; Storage route binding: Bind the target storage address, table / collection name and partition / shard key in the mapping configuration to achieve persistence of routing information for "data-model-storage".

[0024] S3: Trigger and execute the automatic storage process based on the data model specification and mapping configuration. This process includes, but is not limited to, the following sub-steps: Automatic storage structure generation: The system parses the model specification and automatically generates the physical structure of the target storage medium; in relational database scenarios, it automatically generates table creation SQL (field definition, primary key, index, partitioning strategy); when the model specification changes, it automatically generates structure change scripts (CREATE / ALTER) and executes them in the order of preset strategies.

[0025] Automatic storage media matching: Detects the target storage environment (available space, permissions, and network connectivity) based on storage adaptation metadata, automatically switches to standby storage and records the switchover log when an anomaly is detected in the primary storage; automatically initializes and manages the connection pool (e.g., sets the maximum number of connections and timeout) to ensure write performance.

[0026] Automatic data writing: Based on triggering conditions (time-triggered / data volume-triggered / event-triggered), data is extracted, transformation rules are invoked for standardization processing, and batch or real-time writing operations are performed according to the mapping configuration, and the writing target is located according to the partition / table rules.

[0027] Automatic verification and error correction of storage results: After writing, the system automatically performs structure verification (verifying the consistency between the physical structure and the model), data verification (comparing the source data and the written data by sampling according to a certain proportion; in the example, a sampling proportion of 20% and an error rate of ≤0.01% are used as the judgment), and integrity verification (primary key uniqueness and foreign key validity). When verification fails or abnormal data occurs, the system will write the exception to the error log and automatically trigger the retry mechanism (e.g., retry up to 3 times). If the retry still fails, an alarm notification will be sent and manual intervention will be supported.

[0028] Example 2: The following example, using the automatic storage of "real-time equipment data" in a coal mine safety monitoring system, illustrates the implementation process of this invention: S1: Constructing Metadata and Model Specifications Data structure metadata: Define the entity "Real-time Device Data", with the row dimension identified as "Device Code" and the column dimension attributes including "Device Unique Identifier" (foreign key), "Device Code" (nvarchar(50)), "Device Address" (nvarchar(200), Device Type (int), Device Type Name (nvarchar(50)), Unit (nvarhcar(10), Minimum Range (float), Maximum Range (float), Real-time Device Value (float), Real-time Value Time (datetime), Data Status (int)); Constraint metadata: "Maximum range value > Real-time device value > Minimum range value", "Device time ≤ Current time", "Partition by device time"; Storage adaptation metadata: Specify the storage medium as SQL Server 2019, the trigger condition as "trigger batch storage once every 1 minute", the routing information as "IP=192.168.1.100:1433, table name as "AQJK_RealData" (real-time data table for security monitoring information); Generate a specification document containing a metadata dictionary and ER diagrams.

[0029] S2: ETL establishes mapping associations Data source adaptation: The ETL tool automatically connects to the SQL Server metadata configuration database (DB_Meta) and the core business system database (DB_App); Intelligent Mapping: The ETL tool automatically calls the metamodel configuration information for real-time storage services of the security monitoring system from the metadata configuration database (DB_Meta) and loads it into the interface. If the target data body has descriptions related to field names, and the target field names are the same as or similar to the configured field names, a default association is automatically generated, and a manual intervention / modification interface or method is provided. If the target data body does not have field names (using special symbols to separate rows and columns or using field placeholder lengths to distinguish rows and columns), a user configuration mapping interface is provided for manual field configuration mapping. The fields in the metadata configuration are mapped to the target data table as follows: "Device Code → DeviceID", "Device Type → TypeID", "Real-time Value → RealData", "Real-time Value Time → ValTime", "Data Status → Status", "Maximum Range -> Range", "Minimum Range -> Range". Conversion rules: Generate scripts based on the mapping configuration for "maximum range value using the first 2 digits of Range, minimum range value using the last 2 digits of Range", and "ValTime (YYYYMMDD format) to datetime type". Storage route binding: Bind the table partitioning rules, SQL Server table name, partition name, and partition key (device type) in the mapping configuration.

[0030] S3: Automatic Storage Process Storage structure generation: Automatically generate table creation SQL, including table partitioning rules (the initial name of the table in the security monitoring system is AQJK, and the real-time data table rule is RealData); the table partitioning naming rule is to partition by ValTime monthly, the data value time is 2025-11-2, the complete table name is determined to be AQJK_RealData_202511, the partitioning strategy (PARTITIONBYRANGE(TypeID)) and the index (INDEXAQJK_RealData_202511(DeviceID)); Media matching: Detect available space in the SQL Server primary database (≥10GB) and initialize the connection pool (maximum number of connections 30). Automatic write: Triggered once per minute, the ETL tool extracts nearly 1 minute of real-time security monitoring data, transforms it, and writes it in batches to the corresponding partition of the corresponding table; Automatic verification: Extract 20% of the data to verify the accuracy of real-time data and time, check the validity of the "Device ID" foreign key association, and write abnormal data (such as exceeding the threshold value) to the error log and trigger a retry.

[0031] Through the above implementation, the fully automated processing of coal mine monitoring data from model definition to storage was achieved, with storage efficiency improved by 60% and data consistency reaching 100%.

[0032] Example 3: An electronic device, comprising a memory and a processor; The memory is used to store computer programs; The processor is configured to implement the method described in Embodiment 1 when executing the computer program.

[0033] Example 4: A computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described in Embodiment 1.

[0034] Example 5: A computer program product includes a computer program that, when executed by a processor, implements the method described in Example 1.

[0035] In the above embodiments, the reference to "this embodiment" in the specification indicates that a specific feature, structure, or characteristic described in connection with the embodiment is included in at least some embodiments, but not necessarily all embodiments. Multiple appearances of "this embodiment" do not necessarily refer to the same embodiment.

[0036] In the above embodiments, although the invention has been described in conjunction with specific embodiments thereof, many substitutions, modifications, and variations of these embodiments will be apparent to those skilled in the art from the foregoing description. For example, other memory structures (e.g., dynamic RAM (DRAM)) may be used with the embodiments discussed. The embodiments of the invention are intended to cover all such substitutions, modifications, and variations falling within the broad scope of the appended claims.

[0037] As will be understood by those skilled in the art, the computer-readable storage medium described in this embodiment allows for the implementation of all or part of the steps in the above method embodiments by computer program-related hardware. The aforementioned computer program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0038] The electronic terminal provided in this embodiment includes a processor, a memory, a transceiver, and a communication interface. The memory and the communication interface are connected to the processor and the transceiver and complete communication between them. The memory is used to store computer programs, the communication interface is used to perform communication, and the processor and the transceiver are used to run the computer programs, so that the electronic terminal performs the steps of the above method.

[0039] In this embodiment, the memory may include random access memory (RAM) and may also include non-volatile memory, such as at least one disk storage device.

[0040] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0041] This invention can be used in a wide range of general-purpose or special-purpose computing system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices, etc.

[0042] This invention can be described in the general context of computer-executable instructions, such as program modules, that are executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform a specific task or implement a specific abstract data type. This invention can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0043] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A metadata-driven method for automatic storage of two-dimensional standard data, characterized in that: Includes the following steps: S1: Construct a metadata system and define a two-dimensional standard data model specification, and generate a standardized data model specification document. The metadata system includes data structure metadata, constraint metadata and storage adaptation metadata. The model specification clarifies the row and column dimension rules of two-dimensional data and the automatic storage triggering conditions. S2: Establish a dynamic mapping relationship between the target data and the data model specification through ETL tools, and generate a mapping configuration that includes field mapping relationships, transformation rules and storage routing information; S3: Based on the data model specification and mapping configuration, trigger the automatic storage process, including automatically parsing the model specification to generate the storage structure, automatically matching the storage medium, and automatically performing data writing and verification to realize the automatic storage of two-dimensional data.

2. The metadata-driven automatic storage method for two-dimensional standard data according to claim 1, characterized in that: In step S1, the data structure metadata definition includes entity name, row dimension identifier, column dimension attribute and row-column association rules; the constraint metadata definition includes data validation rules and storage constraint rules; the storage adaptation metadata definition includes storage medium type, storage engine parameters and automatic storage trigger conditions. It also generates a standardized model specification document that includes a metadata dictionary, a two-dimensional data model ER diagram, and an automatic storage rule list.

3. The metadata-driven automatic storage method for two-dimensional standard data according to claim 1, characterized in that: In step S1, the automatic storage triggering conditions include three methods: time-triggered, data volume-triggered, and event-triggered. Time-triggered storage is automatically triggered at preset time intervals, data volume-triggered storage is triggered when the data volume reaches a preset threshold, and event-triggered storage is triggered when a data source change event is detected.

4. The metadata-driven automatic storage method for two-dimensional standard data according to claim 1, characterized in that: In step S2, establishing a dynamic mapping relationship between the target data and the data model specification using ETL tools includes: Data source adaptation: Use ETL tools to automatically identify and adapt the target data source type and configure connection parameters; Intelligent field mapping: Automatically generates initial field mappings based on field name similarity and data type consistency, and supports manual intervention and adjustment; Conversion rule configuration: Automatically generate conversion scripts such as type conversion, format standardization, and unit conversion based on model specifications and write them into the mapping configuration; For data without field names, field matching is performed manually, and the field mapping is written into the mapping configuration.

5. The metadata-driven automatic storage method for two-dimensional standard data according to claim 1, characterized in that: In step S2, the conversion rules include type conversion, format standardization, and unit conversion rules; the storage routing information includes the target storage medium address, storage object identifier, and partition / shard key.

6. The metadata-driven automatic storage method for two-dimensional standard data according to claim 1, characterized in that: In step S3, the automatic storage process includes: S31: Automatic storage structure generation: Parses the data model specification and automatically generates the physical structure of the target storage medium; in relational database scenarios, it automatically generates table creation SQL; when a change in the model specification is detected, it automatically generates a structure change script and updates the storage medium synchronously. S32: Automatic storage media matching: Automatically detects the target storage environment based on storage adaptation metadata, automatically switches to backup storage and records the switchover log when the primary storage fails; manages connection pool parameters to ensure write performance; S33: Automatic data writing: Performs data extraction, transformation and writing according to mapping configuration and trigger conditions, supports batch writing and real-time writing, and locates the writing target according to partition / table rules; S34: Automatic verification and error correction of storage results: After writing is completed, structure verification, data verification and integrity verification are automatically performed; structure verification is used to verify the consistency between the storage structure and the model specification, data verification verifies the consistency between the written data and the source data through sampling comparison, and integrity verification verifies the uniqueness of the primary key and the validity of the foreign key association; when the verification fails or abnormal data occurs, the exception is automatically written to the error log and the retry mechanism is triggered. If the retry fails, an alarm notification is issued.

7. The metadata-driven automatic storage method for two-dimensional standard data according to claim 6, characterized in that: In step S31, when a change in model specification is detected, a change script is generated and verified in the test / pre-release environment first. After the verification is passed, the structure update is automatically executed sequentially in the production environment.

Images