Data storage method, device, equipment and storage medium

Abstract

The application relates to the technical field of data transmission, in particular to a data storage method, device and equipment and a storage medium, which comprises the following steps: obtaining to-be-stored operation and maintenance data when a data storage instruction is received, and obtaining to-be-stored configuration data and to-be-stored index data according to the to-be-stored operation and maintenance data; integrating the to-be-stored configuration data and the to-be-stored index data to obtain integrated to-be-stored configuration data and integrated to-be-stored index data; performing format conversion on the integrated to-be-stored configuration data and the integrated to-be-stored index data through a preset field mapping rule; and storing the converted to-be-stored configuration data and the converted to-be-stored index data. Since the to-be-stored configuration data and the to-be-stored index data are integrated first, and then the integrated to-be-stored configuration data and the integrated to-be-stored index data are converted through the preset field mapping rule, the storage management capability of the operation and maintenance configuration data and the operation and maintenance index data can be improved, and the storage efficiency is improved.

Description

Technical Field

[0001] This invention relates to the field of data transmission technology, and in particular to a data storage method, apparatus, device, and storage medium. Background Technology

[0002] Currently, in the cloud environment, the number of nodes and instances of infrastructure, cloud services, and applications far exceeds that of traditional architectures. These changes are also more frequent, and the relationships between them are more complex. As a result, the operation and maintenance configuration data and operation and maintenance indicator data in the cloud environment are diverse, complex, volatile, and massive. Consequently, the storage and management of operation and maintenance configuration data and operation and maintenance indicator data are quite difficult.

[0003] The above content is only used to help understand the technical solution of the present invention and does not represent an admission that the above content is prior art. Summary of the Invention

[0004] The main objective of this invention is to provide a data storage method, apparatus, device, and storage medium, aiming to solve the technical problem in the prior art where the storage and management of operation and maintenance configuration data and operation and maintenance indicator data are difficult due to their diversity, complexity, variability, and massive volume.

[0005] To achieve the above objectives, the present invention provides a data storage method, the method comprising the following steps:

[0006] Upon receiving a data storage instruction, the system acquires the maintenance data to be stored and obtains the configuration data and indicator data to be stored based on the maintenance data to be stored.

[0007] The configuration data to be stored and the indicator data to be stored are integrated to obtain integrated configuration data to be stored and integrated indicator data to be stored.

[0008] The integrated configuration data to be stored and the integrated indicator data to be stored are format converted using preset field mapping rules;

[0009] The converted configuration data and the converted metric data to be stored are stored.

[0010] Optionally, after the step of converting the format of the integrated configuration data to be stored and the integrated indicator data to be stored according to a preset field mapping rule, the method further includes:

[0011] The converted configuration data to be stored is processed according to the preset data relationship rules to obtain the data relationship and data model relationship corresponding to the converted configuration data to be stored.

[0012] Accordingly, the step of storing the converted configuration data to be stored and the converted indicator data to be stored includes:

[0013] The data relationships, the model relationships, and the transformed configuration data to be stored are stored.

[0014] Optionally, the step of storing the data relationship, the model relationship, and the transformed configuration data to be stored includes:

[0015] The data relationship, the model relationship, and the transformed configuration data to be stored are written into a preset single-point database and a preset graph database.

[0016] Accordingly, after the step of storing the data relationship, the model relationship, and the transformed configuration data to be stored, the method further includes:

[0017] Upon receiving a single-point query request, a query is performed using the preset single-point database;

[0018] Upon receiving a relationship query request, a query is performed using the preset graph database.

[0019] Optionally, before the step of processing the transformed configuration data to be stored according to preset data relationship rules to obtain the data relationship and data model relationship corresponding to the transformed configuration data to be stored, the method further includes:

[0020] The converted configuration data to be stored is subjected to quality testing according to preset quality judgment rules;

[0021] The configuration data to be stored is adjusted based on the test results.

[0022] Optionally, after the step of converting the format of the integrated configuration data to be stored and the integrated indicator data to be stored according to a preset field mapping rule, the method further includes:

[0023] The converted index data to be stored is aggregated according to preset compression rules;

[0024] Accordingly, the step of storing the converted configuration data to be stored and the converted indicator data to be stored includes:

[0025] Store the aggregated metric data.

[0026] Optionally, after the step of integrating the configuration data to be stored and the indicator data to be stored to obtain integrated configuration data to be stored and integrated indicator data to be stored, the method further includes:

[0027] Obtain the configuration data source and configuration data characteristics corresponding to the configuration data to be stored, and obtain the indicator data source and indicator data characteristics corresponding to the indicator data to be stored;

[0028] The integrated configuration data to be stored is enriched with fields based on the configuration data source and the configuration data characteristics.

[0029] The integrated indicator data to be stored is enriched with fields based on the source and characteristics of the indicator data.

[0030] Accordingly, the step of converting the format of the integrated configuration data to be stored and the integrated indicator data to be stored using preset field mapping rules includes:

[0031] The format of the enriched configuration data and the enriched indicator data to be stored is converted by using preset field mapping rules.

[0032] Optionally, after the step of storing the converted configuration data to be stored and the converted indicator data to be stored, the method further includes:

[0033] Obtain the business scenario to be applied, and process the stored configuration data and the stored indicator data according to the business scenario to be applied to obtain the data table corresponding to the business scenario to be applied.

[0034] The data table is used to apply the business scenarios to be applied.

[0035] Furthermore, to achieve the above objectives, the present invention also proposes a data storage device, the device comprising:

[0036] The data acquisition module is used to acquire the operation and maintenance data to be stored when a data storage instruction is received, and to obtain the configuration data and indicator data to be stored based on the operation and maintenance data to be stored.

[0037] The data integration module is used to integrate the configuration data to be stored and the indicator data to be stored to obtain the integrated configuration data to be stored and the integrated indicator data to be stored.

[0038] The format conversion module is used to convert the format of the integrated configuration data to be stored and the integrated indicator data to be stored according to preset field mapping rules;

[0039] The data storage module is used to store the converted configuration data and the converted indicator data.

[0040] Furthermore, to achieve the above objectives, the present invention also proposes a data storage device, the device comprising: a memory, a processor, and a data storage program stored on the memory and executable on the processor, the data storage program being configured to implement the steps of the data storage method described above.

[0041] In addition, to achieve the above objectives, the present invention also proposes a storage medium storing a data storage program, which, when executed by a processor, implements the steps of the data storage method described above.

[0042] This invention, upon receiving a data storage instruction, acquires the maintenance data to be stored, and obtains configuration data and indicator data to be stored based on the maintenance data. It then integrates the configuration data and indicator data to obtain integrated configuration data and indicator data. Finally, it performs format conversion on the integrated configuration data and indicator data using a preset field mapping rule, and stores the converted configuration data and indicator data. Because this invention integrates the configuration data and indicator data beforehand and performs format conversion using a preset field mapping rule, it improves the storage management capabilities and storage efficiency of maintenance configuration data and indicator data compared to existing methods. Attached Figure Description

[0043] Figure 1 This is a schematic diagram of the data storage device structure of the hardware operating environment involved in the embodiments of the present invention;

[0044] Figure 2 This is a flowchart illustrating the first embodiment of the data storage method of the present invention;

[0045] Figure 3 This is a data flow diagram of the operation and maintenance data platform in the data storage method of the present invention;

[0046] Figure 4 This is a flowchart illustrating the second embodiment of the data storage method of the present invention;

[0047] Figure 5 This is a flowchart illustrating the third embodiment of the data storage method of the present invention;

[0048] Figure 6 This is a schematic diagram of the functional modules of the data storage method of the present invention, which is a data warehouse for operation and maintenance.

[0049] Figure 7 This is a structural block diagram of the first embodiment of the data storage device of the present invention.

[0050] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0051] It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the invention.

[0052] Reference Figure 1 , Figure 1 This is a schematic diagram of the data storage device structure of the hardware operating environment involved in the embodiments of the present invention.

[0053] like Figure 1 As shown, the data storage device may include: a processor 1001, such as a central processing unit (CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. The communication bus 1002 is used to enable communication between these components. The user interface 1003 may include a display screen or an input unit such as a keyboard; optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wireless-Fidelity (Wi-Fi) interface). The memory 1005 may be high-speed random access memory (RAM) or stable non-volatile memory (NVM), such as a disk drive. The memory 1005 may also optionally be a storage device independent of the aforementioned processor 1001.

[0054] Those skilled in the art will understand that Figure 1 The structure shown does not constitute a limitation on the data storage device and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0055] like Figure 1 As shown, the memory 1005, which serves as a storage medium, may include an operating system, a network communication module, a user interface module, and a data storage program.

[0056] exist Figure 1In the data storage device shown, the network interface 1004 is mainly used for data communication with the network server; the user interface 1003 is mainly used for data interaction with the user; the processor 1001 and the memory 1005 in the data storage device of the present invention can be set in the data storage device, and the data storage device calls the data storage program stored in the memory 1005 through the processor 1001 and executes the data storage method provided in the embodiment of the present invention.

[0057] This invention provides a data storage method, with reference to... Figure 2 , Figure 2 This is a flowchart illustrating the first embodiment of the data storage method of the present invention.

[0058] In this embodiment, the data storage method includes the following steps:

[0059] Step S10: Upon receiving a data storage instruction, obtain the operation and maintenance data to be stored, and obtain the configuration data and indicator data to be stored based on the operation and maintenance data to be stored.

[0060] It should be noted that the method in this embodiment can be applied to scenarios where configuration data and indicator data are stored, or other scenarios requiring data storage. The executing entity in this embodiment can be a data storage device with data processing, network communication, and program execution functions, such as a server, or other devices capable of performing similar or identical functions. This embodiment and the following embodiments will be specifically described using the aforementioned data storage device (hereinafter referred to as the device).

[0061] Understandably, operational data typically includes configuration data, system and application metrics data, log and error information data, and alarm and monitoring information data. Configuration data may include configuration information and parameter data for objects such as infrastructure, cloud services, and applications. Metric data may include, for example, CPU utilization, disk space, memory usage, and network bandwidth. Log and error information data may include system fault logs, security logs, and access logs. Alarm and monitoring information data may include, for example, data related to hardware failures, network congestion, and application performance degradation. The aforementioned operational data can be the operational data to be stored in this embodiment, the aforementioned configuration data can be the aforementioned configuration data to be stored in this embodiment, and the aforementioned metric data can be the aforementioned metric data to be stored in this embodiment.

[0062] It should be understood that the aforementioned operational data to be stored can be obtained through automated recording and collection within the IT infrastructure. It can be used to provide operational personnel with real-time monitoring and management of system operation, and can also provide important basis for the construction of scenarios such as monitoring and emergency response, operational automation, and resource governance.

[0063] In a specific implementation, when the above-mentioned device receives an instruction to perform data storage operations, it can access the maintenance data to be stored from an external data source and extract the configuration data and indicator data to be stored based on the maintenance data to be stored.

[0064] Step S20: Integrate the configuration data to be stored and the indicator data to be stored to obtain integrated configuration data to be stored and integrated indicator data to be stored.

[0065] It should be noted that the above-mentioned device can integrate the obtained configuration data and indicator data to be stored into the data source inside the operation and maintenance data platform. In this embodiment, the above-mentioned device can support data integration from external data sources such as Kafka, MySQL, and API.

[0066] Understandably, the aforementioned equipment can establish a unified data system for storing indicator data and configuration data, and build an operation and maintenance data warehouse based on the unified data system. This warehouse may include a data integration layer, a data processing layer, a data import layer, and a data service layer. The aforementioned data integration layer can be used to access source data (i.e., the aforementioned operation and maintenance data to be stored), and integrate the configuration data to be stored and the operation and maintenance data to be stored within it through Kafka, MySQL, and APIs, and synchronize them to the data source within the platform.

[0067] It should be understood that references are available. Figure 3 To explain, Figure 3 This is a data flow diagram of the operation and maintenance data platform in the data storage method of the present invention, such as... Figure 3 As shown, Figure 3 The example demonstrates the use of external data sources such as ElasticSearch, ClickHouse, HTTP, Hive, MySQL, and Kafka. Other methods are also possible, and this example does not impose any limitations on them. Figure 3 Real-time synchronization in this context refers to synchronizing configuration data and maintenance data to be stored to the data source within the platform.

[0068] It is important to emphasize that when the configuration data to be stored is integrated into the platform's internal data source, the aforementioned devices can also write the configuration data to be stored into both the Kafka streaming table and the original MySQL table without changing the original data content. This ensures the preservation of the original configuration data to be stored and facilitates troubleshooting of subsequent data issues.

[0069] It should also be emphasized that when the metric data to be stored is integrated into the platform's internal data source, the aforementioned devices can also write the metric data to be stored into the Kafka flow table without changing the original data content, thus ensuring the preservation of the original metric data to be stored.

[0070] In practical implementation, the above-mentioned device can integrate the configuration data to be stored and the indicator data to be stored through the data integration function to obtain the integrated configuration data and the integrated indicator data to be stored.

[0071] Step S30: Convert the format of the integrated configuration data to be stored and the integrated indicator data to be stored by means of a preset field mapping rule.

[0072] It should also be noted that when the aforementioned configuration data and indicator data to be stored are integrated into the platform's internal data source, the configuration data to be stored is synchronized from the data integration layer to the data processing layer through the data synchronization function, and the indicator data to be stored is synchronized from the data integration layer to the data processing layer through the data processing function. The aforementioned data processing layer can be used to process detailed data, that is, to perform format conversion on the integrated configuration data and integrated indicator data to be stored through the aforementioned preset field mapping rules, converting the format of the integrated configuration data and integrated indicator data to be stored into the format of the reviewed model design.

[0073] It is understood that the above-mentioned preset field mapping rules can be set according to the actual situation, and the format of the above-mentioned reviewed model design can also be set by the user. This embodiment does not impose any restrictions on these aspects.

[0074] Furthermore, to facilitate subsequent querying and management, in this embodiment, after step S20, the following step is also included:

[0075] Step S21: Obtain the configuration data source and configuration data characteristics corresponding to the configuration data to be stored, and obtain the indicator data source and indicator data characteristics corresponding to the indicator data to be stored.

[0076] It should be understood that the source of the configuration data corresponding to the above-mentioned configuration data to be stored and the source of the indicator data to be stored can both prove the source of the data. They can be obtained through manual input, sensor collection and interfaces provided by external data sources. Of course, they can also be obtained through other means, which are not limited in this embodiment.

[0077] It should be noted that the configuration data characteristics corresponding to the configuration data to be stored and the indicator data characteristics corresponding to the indicator data to be stored can both reflect the characteristics of the data, such as the data type (numerical, text, etc.), the data format (structured, semi-structured or unstructured), and the data attributes (size, unit, etc.). This embodiment does not impose any restrictions on these.

[0078] Step S22: Enrich the fields of the integrated configuration data to be stored according to the configuration data source and the configuration data characteristics;

[0079] Step S23: Enrich the fields of the integrated indicator data to be stored according to the source and characteristics of the indicator data.

[0080] Understandably, the aforementioned devices can add additional fields to the configuration data and indicator data to be stored, based on the data source and data characteristics, thereby enriching the expressive power and flexibility of the configuration data and indicator data to be stored.

[0081] Accordingly, step S30 includes:

[0082] Step S31: Convert the format of the enriched configuration data to be stored and the enriched indicator data to be stored by using preset field mapping rules.

[0083] It should be understood that after enriching the field data content, the format of the data to be stored configuration data and the data to be stored indicator data can be converted, which will facilitate subsequent querying of data sources and management.

[0084] In a specific implementation, the aforementioned device can enrich the fields of the configuration data to be stored and the indicator data to be stored, and then convert the format of the enriched configuration data to be stored and the enriched indicator data to be stored through a preset field mapping rule.

[0085] Step S40: Store the converted configuration data and the converted indicator data.

[0086] It should be noted that after the data processing layer processes the data, it can be stored in the data import layer. The aforementioned devices can use the data import function to import the configuration data to be stored into the database via a unified data import program. They can also use the data processing function to store the indicator data to be stored. Figure 3 Storage within.

[0087] Understandably, continuing as Figure 3 As shown, after the configuration data and metric data to be stored are synchronized, the data stream can be transmitted through Kafka and MQ (message queue), that is, the configuration data and metric data to be stored can be transmitted and the task can be processed. Real-time tasks and offline tasks can be processed, and the processing results can be stored.

[0088] It should be understood that continuing as Figure 3 As shown, in data synchronization, this embodiment can also support offline full, full, and Change Data Capture (CDC) synchronization methods, and can generate metadata synchronously to achieve multi-table synchronization and full database synchronization.

[0089] Furthermore, to facilitate subsequent processing, in this embodiment, after step S40, the following step is also included:

[0090] Step S50: Obtain the business scenario to be applied, and process the stored configuration data to be stored and the stored indicator data to be stored according to the business scenario to be applied to obtain the data table corresponding to the business scenario to be applied.

[0091] Step S60: Apply the business scenario to be applied through the data table.

[0092] It is understood that the aforementioned application scenarios can be any scenarios where the aforementioned configuration data and indicator data to be stored can be applied. In this embodiment, the aforementioned device can store the configuration data and indicator data to be stored from the data import layer to the data service layer for further processing to obtain a data table that meets the application scenarios, which can then be applied to the application scenarios, such as multi-table association of configuration data, exploration of historical data; aggregation calculation and analysis calculation of indicator data; fusion wide table calculation and materialized view of configuration data and indicator data; view calculation of configuration data and indicator data, etc. This embodiment does not impose any specific limitations.

[0093] In practice, the aforementioned device can acquire the application scenario and further process the storage configuration data and storage index data according to the application scenario, and apply the obtained data table to the application scenario, thereby facilitating subsequent query and other operations.

[0094] In this embodiment, when the device receives an instruction to perform data storage operations, it can access the maintenance data to be stored from an external data source and extract the configuration data and indicator data to be stored based on the maintenance data. It can enrich the fields of the configuration data and indicator data, and then convert the format of the enriched data using preset field mapping rules. The converted configuration data and indicator data are then stored. Because this embodiment can first integrate the configuration data and indicator data, and then convert their format using preset field mapping rules, it can improve the storage management capabilities of maintenance configuration data and maintenance indicator data. Furthermore, because this embodiment is compatible with the processing of configuration data and indicator data, it reduces data management costs. It fully considers the diversity, volume, variability, and complexity of configuration data in the cloud environment to support the business's requirements for the accuracy and timeliness of configuration data; and it fully considers the diversity, massive volume, and high density of indicator data in the cloud environment to support the business's historical dependence on and real-time requirements for indicator data.

[0095] refer to Figure 4 , Figure 4 This is a flowchart illustrating the second embodiment of the data storage method of the present invention.

[0096] To further facilitate subsequent management and retrieval, such as Figure 4 As shown, in this embodiment, after step S30, the following step is also included:

[0097] Step S32: Process the converted configuration data to be stored according to the preset data relationship rules to obtain the data relationship and data model relationship corresponding to the converted configuration data to be stored.

[0098] Accordingly, step S40 above includes:

[0099] Step S41: Store the data relationship, the model relationship, and the transformed configuration data to be stored.

[0100] It should be noted that there may be certain relationships between the configuration data to be stored, and there may also be certain relationships between the configuration data to be stored and the data model in which they are located. Therefore, in this embodiment, the above-mentioned device can obtain the relationship between the configuration data to be stored and the relationship between the configuration data to be stored and the data model according to the predefined preset data relationship rules, and store the obtained data relationship, model relationship and configuration data to be stored into the data import layer.

[0101] It is understood that the above-mentioned preset data relationship rules can be set according to the actual relationship, and this embodiment does not impose any restrictions on this.

[0102] It should be understood that when accessing the data import layer, data relationships, model relationships, and configuration data to be stored can be written to a database built with MySQL and a graph database via Elasticsearch. Figure 3 The storage includes ElasticSearch, MySQL, and graph databases.

[0103] In a specific implementation, the aforementioned device can determine the relationships between the converted configuration data to be stored according to preset data relationships, obtain the data relationships between the converted configuration data to be stored and the data module relationships with the data model, and write them into the database built by MySQL and the graph database.

[0104] Furthermore, to improve query efficiency, in this embodiment, step S41 includes:

[0105] Step S411: Write the data relationship, the model relationship, and the transformed configuration data to be stored into a preset single-point database and a preset graph database;

[0106] Accordingly, after step S41 above, the following steps are also included:

[0107] Step S412: Upon receiving a single-point query request, perform a query through the preset single-point database;

[0108] Step S413: Upon receiving a relation query request, perform a query through the preset graph database.

[0109] It should also be noted that the aforementioned preset single-point database is the same as the database built using the MySQL database, and the aforementioned preset graph database is the same as the graph database.

[0110] It is understood that the above single-point query request can be a request to query a single piece of data in the preset graph database, and the above relation query request can be a request to query data relations and model relations.

[0111] It should be understood that, as Figure 3 As shown, the aforementioned device can receive the single-point query request and relational query request through the standard API and custom API in the data service. When a single-point query request is received, a single data query can be performed through a database built with MySQL. When a relational query request is received, relational data queries can be performed through a graph database.

[0112] It is important to emphasize that when the configuration data to be stored changes, the changed configuration data can be synchronously updated in Hive, i.e. Figure 3 Hive, stored in the database, can also generate new data tables through data processing jobs to provide data services.

[0113] In a specific implementation, the aforementioned device can write data relationships, model relationships, and the transformed configuration data to be stored into a database built with MySQL and a graph database. Then, when a single-point query request is received, the query is performed in the database built with MySQL, and when a relationship query request is received, the query is performed in the graph database.

[0114] In this embodiment, the device described above can determine the relationships between the converted configuration data to be stored according to preset data relationships, obtain the data relationships between the converted configuration data to be stored and the data module relationships with the data model, and write them to the database built by MySQL and the graph database. At the same time, it can write the data relationships, model relationships and the converted configuration data to be stored to the database built by MySQL and the graph database, so that when a single point query request is received, a query is performed in the database built by MySQL, and when a relationship query request is received, a query is performed in the graph database.

[0115] refer to Figure 5 , Figure 5 This is a flowchart illustrating the third embodiment of the data storage method of the present invention.

[0116] To improve the quality of the configuration data to be stored, such as Figure 5 As shown, in this embodiment, after step S32, the following step is also included:

[0117] Step S33: Perform quality checks on the converted configuration data to be stored according to preset quality judgment rules;

[0118] Step S34: Adjust the configuration data to be stored based on the detection results.

[0119] It should be noted that the above-mentioned preset quality judgment rules can be pre-set rules for judging the quality of the configuration data to be stored, and the specific content of the rules is not limited in this embodiment.

[0120] In practice, the aforementioned device can perform quality rule judgment on the data to be stored according to preset quality judgment rules after format conversion, obtain the detection results, and adjust the data to be stored based on the detection results to promptly process and improve the quality of the data to be configured.

[0121] Furthermore, to reduce data redundancy, in this embodiment, after step S30, the following step is also included:

[0122] Step S35: Perform dimensional aggregation on the converted index data to be stored according to preset compression rules;

[0123] Accordingly, step S40 includes:

[0124] Step S42: Store the aggregated index data to be stored.

[0125] It is understandable that the above-mentioned preset compression rules can be rules with certain dimensions. The above-mentioned preset compression rules can be generated by defining dimensions, and then the preset compression rules can be used to aggregate the dimensions of the indicator data to be stored, thereby compressing the high-density indicator data to be stored.

[0126] It should be understood that the aforementioned devices can also use data processing functions to write the obtained metric data to be stored into ClickHouse and Hive via ElasticSearch. Furthermore, if the metric data to be stored changes, the changed metric data can also be synchronously updated in Hive. Figure 3 ClickHouse and Hive are stored in the middle.

[0127] It should also be noted that during synchronous updates, the aforementioned devices can simultaneously create materialized views or views to provide data query services, and provide data services through data processing operations. Figure 3 The data service utilizes both standard and custom APIs.

[0128] In practice, the aforementioned devices can perform dimensional aggregation on the index data to be stored according to preset compression rules, obtain compressed index data to be stored, and store the compressed index data to be stored in ClickHouse and Hive.

[0129] Furthermore, to facilitate understanding of all the functions of the aforementioned operations and maintenance data warehouse, refer to... Figure 6 , Figure 6 This is a schematic diagram of the functional modules of the data storage method of the present invention, namely, the operation and maintenance data warehouse. Figure 6 As shown, in this embodiment, the capabilities can be divided into four modules, including: asset configuration module, data modeling module, data development module, and data storage module.

[0130] Understandably, the functions of the aforementioned asset configuration module may include: model design, data synchronization, data import, unified terminology, data access control, and logging. The model design function may include: the data source manager designing asset models, fields, and relationships based on the data source's structure and the user's data requirements. Model metadata may include business, technical, and process metadata, as well as the model's quality rules. Field metadata may include the field's name, attributes, whether it references unified terminology, and the field's quality rules. Relationship metadata may include relationships between models, such as joins, derivations, and references. The data synchronization function may include: mapping and calculating asset data records according to defined preset field mapping rules and quality detection rules, and storing the data in the data import layer. The data import function may include: importing data from the data import layer into a database built with MySQL and a graph database. The unified terminology function may include: defining unified terminology and basic data, and managing the references to unified terminology and basic data. The data access control function may include: managing data access permissions. The logging function may include: managing all logs of the asset configuration module, specifically including version information for asset design, logs for synchronization tasks, logs for the data import function, and version information for unified terminology design.

[0131] The data modeling module described above includes the following functions: data warehouse planning, dimensional modeling, data standardization, and data metrics. The data warehouse planning function includes designing the data integration layer, data processing layer, data import layer, and data service layer, while simultaneously managing the data within these layers. The dimensional modeling function includes enabling data source owners to design asset models, fields, and relationships—three types of metadata—based on the data source's structure and the user's data needs. The data standardization function includes managing the metric name dictionary and standardizing terminology and basic data references.

[0132] The data development module described above includes the following functions: data integration, real-time development, offline development, data querying, UDF management, and task management. The data integration function includes enriching certain fields within an internal data source without altering existing data from an external data source. The real-time and offline development functions support multiple development modes and languages, such as FlinkSQL, Flink JAR packages, HiveSQL, and Python scripts, and include monitoring and scheduling management of processing tasks.

[0133] The database types in the aforementioned data storage modules can include: MySQL, graph databases, ClickHouse, and Hive. Data to be configured is stored in databases built with MySQL and graph databases, supporting relational and single-point queries on the configuration data, and synchronized to Hive. Metric data to be stored can be stored in ClickHouse and synchronized to Hive. Synchronizing configuration and metric data to Hive enables more complex data analysis scenarios, such as multi-table joins and historical data exploration of configuration data; aggregation and analysis calculations of metric data; fusion wide table calculations and materialized views of configuration and metric data; and view calculations (i.e.,...) of configuration and metric data. Figure 6 (Historical data management in China).

[0134] It should be emphasized that the historical data storage rules in this embodiment can be as follows: the database built by default MySQL and the graph database store the latest day's data; ClickHouse stores data for the past 7 days, the past month's data on Mondays, and the past year's data on the 1st of each month by default; the data in the MySQL database and the graph database are automatically synchronized to Hive, where the data for the past 7 days, the past month's data on Mondays, and the past year's data on the 1st of each month are stored; Hive stores minute-level data for the past month, detailed data for the 1st of each month for the past year, ten-minute-level data for the past quarter, and hour-level data for the past year by default; of course, other historical data storage rules can also be used, and this embodiment does not limit them.

[0135] The device described in this embodiment can, after format conversion, perform quality rule judgment on the data to be stored according to preset quality judgment rules, obtain detection results, and adjust the data to be stored based on the detection results to promptly process and improve the quality of the data to be configured; it can also perform dimensional aggregation on the data to be stored according to preset compression rules to obtain compressed data to be stored, and store the compressed data to be stored in ClickHouse and Hive.

[0136] It should also be emphasized that, by establishing a standardized process for accessing, processing, storing, and serving configuration data and indicator data to be stored, this embodiment can reduce the semantic diversity of data model definitions for different data owners, reduce the standardization of data processing tasks for different data developers, and reduce ambiguity and misunderstanding of data consumption by different data consumers. This reduces the problems of data fragmentation and lack of integration, and builds a unified definition, system, and architecture for operation and maintenance data, allowing data to realize its value.

[0137] Furthermore, embodiments of the present invention also propose a storage medium storing a data storage program, which, when executed by a processor, implements the steps of the data storage method described above.

[0138] In addition, refer to Figure 7 , Figure 7 This is a structural block diagram of a first embodiment of the data storage device of the present invention. The present invention also proposes a data storage device, which includes:

[0139] The data acquisition module 701 is used to acquire the operation and maintenance data to be stored when a data storage instruction is received, and to obtain the configuration data to be stored and the indicator data to be stored based on the operation and maintenance data to be stored.

[0140] The data integration module 702 is used to integrate the configuration data to be stored and the indicator data to be stored to obtain the integrated configuration data to be stored and the integrated indicator data to be stored.

[0141] The format conversion module 703 is used to convert the format of the integrated configuration data to be stored and the integrated indicator data to be stored according to a preset field mapping rule;

[0142] The data storage module 704 is used to store the converted configuration data to be stored and the converted indicator data to be stored.

[0143] In this embodiment, when the device receives an instruction to perform data storage operations, it can access the maintenance data to be stored from an external data source and extract the configuration data and indicator data to be stored based on the maintenance data. It can enrich the fields of the configuration data and indicator data, and then convert the format of the enriched data using preset field mapping rules. The converted configuration data and indicator data are then stored. Because this embodiment can first integrate the configuration data and indicator data, and then convert their format using preset field mapping rules, it can improve the storage management capabilities of maintenance configuration data and maintenance indicator data. Furthermore, because this embodiment is compatible with the processing of configuration data and indicator data, it reduces data management costs. It fully considers the diversity, volume, variability, and complexity of configuration data in the cloud environment to support the business's requirements for the accuracy and timeliness of configuration data; and it fully considers the diversity, massive volume, and high density of indicator data in the cloud environment to support the business's historical dependence on and real-time requirements for indicator data.

[0144] Other embodiments or specific implementations of the data storage device of the present invention can be referred to the above-described method embodiments, and will not be repeated here.

[0145] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.

[0146] The sequence numbers of the above embodiments of the present invention are merely for description and do not represent the superiority or inferiority of the embodiments. Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as read-only memory / random access memory, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0147] The above are merely preferred embodiments of the present invention and do not limit the scope of the patent. Any equivalent structural or procedural transformations made based on the description and drawings of the present invention, or direct or indirect applications in other related technical fields, are similarly included within the scope of patent protection of the present invention.

Claims

1. A data storage method, characterized in that, The data storage method includes the following steps: Upon receiving a data storage instruction, the system acquires the maintenance data to be stored and obtains the configuration data and indicator data to be stored based on the maintenance data to be stored. The configuration data to be stored consists of the configuration information and parameter data of the object. The configuration data to be stored and the indicator data to be stored are integrated to obtain integrated configuration data to be stored and integrated indicator data to be stored. The integration involves accessing the source data and synchronizing the configuration data to be stored and the indicator data to be stored to the platform's internal data source. The integrated configuration data to be stored and the integrated indicator data to be stored are format converted using preset field mapping rules; The converted configuration data and the converted metric data to be stored are stored. After the step of converting the format of the integrated configuration data to be stored and the integrated indicator data to be stored using preset field mapping rules, the method further includes: The converted configuration data to be stored is processed according to the preset data relationship rules to obtain the data relationship and data model relationship corresponding to the converted configuration data to be stored. Accordingly, the step of storing the converted configuration data to be stored and the converted indicator data to be stored includes: The data relationship, the model relationship, and the transformed configuration data to be stored are written into a preset single-point database and a preset graph database. When a single-point query request is received, the query is performed through the preset single-point database; when a relationship query request is received, the query is performed through the preset graph database.

2. The data storage method as described in claim 1, characterized in that, Before the step of processing the converted configuration data to be stored according to preset data relationship rules to obtain the data relationship and data model relationship corresponding to the converted configuration data to be stored, the method further includes: The converted configuration data to be stored is subjected to quality testing according to preset quality judgment rules; The configuration data to be stored is adjusted based on the test results.

3. The data storage method as described in claim 1, characterized in that, After the step of converting the format of the integrated configuration data to be stored and the integrated indicator data to be stored using preset field mapping rules, the method further includes: The converted index data to be stored is aggregated according to preset compression rules; Accordingly, the step of storing the converted configuration data to be stored and the converted indicator data to be stored includes: Store the aggregated metric data.

4. The data storage method according to any one of claims 1 to 3, characterized in that, After the step of integrating the configuration data to be stored and the indicator data to be stored to obtain the integrated configuration data to be stored and the integrated indicator data to be stored, the method further includes: Obtain the configuration data source and configuration data characteristics corresponding to the configuration data to be stored, and obtain the indicator data source and indicator data characteristics corresponding to the indicator data to be stored; The integrated configuration data to be stored is enriched with fields based on the configuration data source and the configuration data characteristics. The integrated indicator data to be stored is enriched with fields based on the source and characteristics of the indicator data. Accordingly, the step of converting the format of the integrated configuration data to be stored and the integrated indicator data to be stored using preset field mapping rules includes: The format of the enriched configuration data and the enriched indicator data to be stored is converted by using preset field mapping rules.

5. The data storage method as described in claim 1, characterized in that, After the step of storing the converted configuration data and the converted indicator data, the method further includes: Obtain the business scenario to be applied, and process the stored configuration data and the stored indicator data according to the business scenario to be applied to obtain the data table corresponding to the business scenario to be applied. The data table is used to apply the business scenarios to be applied.

6. A data storage device, characterized in that, The device includes: The data acquisition module is used to acquire the operation and maintenance data to be stored when a data storage instruction is received, and to obtain the configuration data to be stored and the indicator data to be stored based on the operation and maintenance data to be stored. The configuration data to be stored is the configuration information of the object and the data corresponding to the parameters. The data integration module is used to integrate the configuration data to be stored and the indicator data to be stored to obtain the integrated configuration data to be stored and the integrated indicator data to be stored. The integration involves accessing the source data and synchronizing the configuration data to be stored and the indicator data to be stored to the platform's internal data source. The format conversion module is used to convert the format of the integrated configuration data to be stored and the integrated indicator data to be stored according to preset field mapping rules; The data storage module is used to store the converted configuration data and the converted indicator data to be stored. The format conversion module is also used to process the converted configuration data to be stored according to preset data relationship rules, so as to obtain the data relationship and data model relationship corresponding to the converted configuration data to be stored; The data storage module is further configured to write the data relationship, the model relationship, and the transformed configuration data to be stored into a preset single-point database and a preset graph database; when a single-point query request is received, a query is performed through the preset single-point database; when a relationship query request is received, a query is performed through the preset graph database.

7. A data storage device, characterized in that, The device includes: a memory, a processor, and a data storage program stored on the memory and executable on the processor, the data storage program being configured to implement the steps of the data storage method as described in any one of claims 1 to 5.

8. A storage medium, characterized in that, The storage medium stores a data storage program, which, when executed by a processor, implements the steps of the data storage method as described in any one of claims 1 to 5.

Images