DataOps platform for realizing big data construction based on DevOPs

Abstract

The invention discloses a DataOps platform for realizing big data construction based on DevOPs, and belongs to the technical field of big data release processing, the DataOps platform comprises an automatic operation and maintenance system, a data rule engine and a data service API, the automatic operation and maintenance system is used for automatically operating and maintaining to-be-released data, the data rule engine is used for normatively preparing metadata and test cases, and the data service API is used for operating and maintaining the to-be-released data. The method is used for preposed data structure, metadata standard conformity and business standard use case conformity. According to the method, the metadata standard and the business standard use case are formed through the automatic operation and maintenance system in the Dataops platform, the data structure, the metadata standard conformity and the business standard use case conformity of the to-be-published data are automatically tested, it is ensured that the to-be-published data meet the data publishing standard of big data, problems existing in the to-be-published data are found in advance, and the data publishing efficiency is improved. Problem data is prevented from being published to the target data warehouse, the publishing quality is guaranteed, and the publishing failure rate is reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of big data publishing and processing, and in particular to a DataOps platform for implementing big data construction based on DevOps. Background Art

[0002] Devops is a general term for a set of processes, methods and systems, including development and operations, and is a general term for development and operation and maintenance. During the software design process, coordination should be carried out between the development department and the operation and maintenance department to ensure the efficient use of various work processes and methods, and to provide a reliable reference for project management. In this process, by building a Dataops platform, data can be directly obtained from the data source in real time, and then enter the data lake. Through tools such as streaming data processing, real-time data warehouses, and large-scale automatic data processing processes, a closed loop of analytical data management is constructed, and ultimately diversified data services are output, thereby reducing the entry threshold and learning complexity for data workers.

[0003] Currently, in big data application scenarios, after data is released, there is a delay in discovering anomalies in the released data, which causes the released data to have an impact on the target data warehouse. When processing these abnormal data, more manpower and resources are required, resulting in long processing cycles, low efficiency, and high costs. Summary of the invention

[0004] The purpose of the present invention is to solve the problem that in the current big data application scenarios, after the data is released, there is a delay in the discovery of the published data anomalies, which causes the published data to have an impact on the target data warehouse. When processing these abnormal data, more manpower and resources need to be invested, and there are problems such as long processing cycle, low efficiency and high cost. A DataOps platform based on DevOPs to realize big data construction is proposed.

[0005] In order to achieve the above object, the present invention adopts the following technical solutions: The DataOps platform for big data construction based on DevOPs is built by data testing stakeholders, who are composed of development and operation teams, data analysis teams, and data testing teams. It includes an automated operation and maintenance system, a data rule engine, and a data service API. The automated operation and maintenance system consists of a data upload module, a metadata preparation module, a test case preparation module, and a test result tracking module, and is used to automatically operate and maintain data to be released. The data rule engine consists of a metadata standard generation module and a business standard case generation module, which are used to standardize the preparation of metadata and test cases; The data service API is provided with a data detection interface to be released, which is used to compare metadata and business data, detect the front-end data structure, metadata standard compliance and business standard use case compliance, and push the detection results to data test stakeholders through the test result tracking module, and publish the data that passes the metadata standard compliance and business standard use case compliance to the target data warehouse.

[0006] Preferably, the process in which the data upload module participates in the automated operation and maintenance system is a test data upload process: the development and operation and maintenance team prepares the data to be tested according to the data structure requirements of the big data, and uploads the data to be released to the automated operation and maintenance system.

[0007] Preferably, the process in which the metadata preparation module participates in the automated operation and maintenance system is a metadata maintenance process: the data analysis team prepares metadata according to metadata management specifications, and regularly maintains and updates the metadata through the automated operation and maintenance system.

[0008] Preferably, the process in which the test case preparation module participates in the automated operation and maintenance system is a test case maintenance process: the data testing team prepares test cases in combination with business data rules and test case writing specifications, and regularly maintains and updates the test cases through the automated operation and maintenance system.

[0009] Preferably, the process in which the test result tracking module participates in the automated operation and maintenance system is a test result improvement process: data test stakeholders track the test results, adjust the data to be released according to the test results, and improve the metadata standards and test case standards.

[0010] Preferably, in the data service API, the steps for detecting the data to be published are as follows: Step S1: Data structure detection: the data service API detects the data structure of the data to be published according to the data structure specification of big data; Step S2: Metadata standard compliance detection: the data service API detects the data to be published according to the metadata standard; Step S3: Business standard use case compliance detection: the data service API detects the data to be published according to the business standard use case; Step S4: Send the data to be published. The data that passes the metadata standard compliance and business standard use case compliance is published to the target data warehouse, and the data that fails the compliance is blocked.

[0011] Compared with the prior art, the present invention has the following beneficial effects: 1. Through the automated operation and maintenance system in the Dataops platform, metadata standards and business standard use cases are formed, and the data structure, metadata standard compliance, and business standard use case compliance of the data to be published are automatically tested to ensure that the data to be published meets the data publishing standards of big data, discover problems with the data to be published in advance, avoid publishing problematic data to the target data warehouse, ensure the quality of publishing, and reduce the failure rate of publishing.

[0012] 2. Through the test result tracking module in the data service API, timely adjustments can be made to the problems of the data to be released during the test process, and the metadata standards and test case standards can be further improved, so as to solve the problem of delayed discovery of problems after data release in big data, resulting in delayed processing, reduce the time when system problems occur, and improve the available time of system business. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 A flowchart of the detection and processing of data to be released by the DataOps platform for big data construction based on DevOps proposed by the present invention; Figure 2 This is a flowchart of the detection process within the data service API of the DataOps platform for implementing big data construction based on DevOps proposed in the present invention. DETAILED DESCRIPTION

[0014] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0015] Example, see Figure 1 to Figure 2 , a DataOps platform based on DevOPs to realize big data construction, with data testing stakeholders participating in the platform construction. Data testing stakeholders are composed of development and operation teams, data analysis teams and data testing teams, including automated operation and maintenance systems, data rule engines and data service APIs. The automated operation and maintenance system consists of data upload modules, metadata preparation modules, test case preparation modules and test result tracking modules, which are used for automated operation and maintenance of data to be released; The data rule engine consists of a metadata standard generation module and a business standard case generation module, which are used to standardize the preparation of metadata and test cases; The data service API is equipped with a data detection interface for comparing metadata and business data, detecting the pre-data structure, metadata standard compliance, and business standard use case compliance, and pushing the detection results to data test stakeholders through the test result tracking module, and publishing data that has passed the metadata standard compliance and business standard use case compliance to the target data warehouse.

[0016] It should be noted that the data operation and maintenance team extracts supply chain finance-related big data information through ETL, and uses the data rule engine to classify and analyze business data, technical data, and management data to form metadata standards; The data analysis team regularly provides metadata maintenance data to improve metadata standards. Business standard use cases are formed by combining supply chain finance-related business rules and manually defined root tables and their value ranges; The data testing team regularly provides test case maintenance data to improve the test case standards. Through the data service API of the automated testing system, the data structure, metadata standard compliance, and business standard case compliance of the data to be published are automatically tested. Through the above automated data testing, it is ensured that the data to be published meets the data publishing standards of big data, and problems with the data to be published are discovered in advance, avoiding the release of problematic data to the target data warehouse, ensuring the quality of the release, and reducing the failure rate of the release; Based on the above, this method solves the problem of delayed discovery of problems after data release in most big data. At the same time, the problem feedback rate after release has been reduced from 1.5% to 0.05%. Taking 500 data releases and 15 minutes to process a problem feedback as an example, the number of feedback problems has been reduced from 7.5 times to 0.25 times, and the problem processing time has been reduced from 112.5 minutes to 3.75 minutes. This greatly reduces the time it takes for system problems to occur and improves the available time of system services.

[0017] Furthermore, the process in which the data upload module participates in the automated operation and maintenance system is the test data upload process: the development and operation and maintenance teams prepare the data to be tested according to the data structure requirements of the big data, and upload the data to be released to the automated operation and maintenance system.

[0018] Furthermore, the process in which the metadata preparation module participates in the automated operation and maintenance system is the metadata maintenance process: the data analysis team prepares metadata according to metadata management specifications, and regularly maintains and updates the metadata through the automated operation and maintenance system.

[0019] It should be noted that: when generating metadata standards, the data rule engine extracts metadata maintenance information at a predetermined frequency, and parses and maintains data through the mate management rule model to improve metadata standards and ensure the accuracy of metadata standards.

[0020] Furthermore, the process in which the test case preparation module participates in the automated operation and maintenance system is the test case maintenance process: the data testing team prepares test cases based on business data rules and test case writing specifications, and regularly maintains and updates the test cases through the automated operation and maintenance system.

[0021] It should be noted that when generating business standard use cases, the data rule engine tests the case maintenance information at a predetermined frequency, parses and processes the test case information, improves the test case standards, and ensures the accuracy of the test case standards.

[0022] Furthermore, the process in which the test result tracking module participates in the automated operation and maintenance system is the test result improvement process: data test stakeholders track the test results, adjust the data to be released based on the test results, and improve the metadata standards and test case standards.

[0023] It should be noted that the results after passing the data service API test can be viewed through the test result tracking module, and the data test results to be released will be pushed to relevant parties (data testing team, data analysis team, development / operation and maintenance team, team leaders, etc.) through enterprise WeChat / DingTalk.

[0024] like Figure 2 As shown, in the data service API, the steps for detecting the data to be published are as follows: Step S1: Data structure detection: the data service API detects the data structure of the data to be published according to the data structure specification of big data; Step S2: Metadata standard compliance detection: the data service API detects the data to be published according to the metadata standard; Step S3: Business standard use case compliance detection: the data service API detects the data to be published according to the business standard use case; Step S4: Send the data to be published. The data that passes the metadata standard compliance and business standard use case compliance is published to the target data warehouse, and the data that fails the compliance is blocked.

[0025] It should be noted that: in step S4, the detection results of the data to be released will be pushed to data test stakeholders through enterprise WeChat / DingTalk, and the test results will be tracked. According to the test results, the data to be released will be adjusted, and the metadata standards and test case standards will be improved, so as to achieve the effect of timely adjustment of the test standards and only deliver data that passes the compliance to the target data warehouse.

[0026] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. The DataOps platform for big data construction based on DevOps is built by data testing stakeholders, who are composed of development and operation teams, data analysis teams, and data testing teams. It is characterized by: It includes an automated operation and maintenance system, a data rule engine and a data service API. The automated operation and maintenance system consists of a data upload module, a metadata preparation module, a test case preparation module and a test result tracking module, and is used for automated operation and maintenance of data to be released; The data rule engine consists of a metadata standard generation module and a business standard case generation module, which are used to standardize the preparation of metadata and test cases; The data service API is provided with a data detection interface to be released, which is used to compare metadata and business data, detect the front-end data structure, metadata standard compliance and business standard use case compliance, and push the detection results to data test stakeholders through the test result tracking module, and publish the data that passes the metadata standard compliance and business standard use case compliance to the target data warehouse.

2. The DataOps platform for implementing big data construction based on DevOps according to claim 1, characterized in that: The process in which the data upload module participates in the automated operation and maintenance system is the test data upload process: the development and operation and maintenance teams prepare the data to be tested according to the data structure requirements of the big data, and upload the data to be released to the automated operation and maintenance system.

3. The DataOps platform for implementing big data construction based on DevOps according to claim 1, characterized in that: The process in which the metadata preparation module participates in the automated operation and maintenance system is the metadata maintenance process: the data analysis team prepares metadata according to metadata management specifications, and regularly maintains and updates the metadata through the automated operation and maintenance system.

4. The DataOps platform for implementing big data construction based on DevOps according to claim 1, characterized in that: The process in which the test case preparation module participates in the automated operation and maintenance system is the test case maintenance process: the data testing team prepares test cases in combination with business data rules and test case writing specifications, and regularly maintains and updates the test cases through the automated operation and maintenance system.

5. The DataOps platform for implementing big data construction based on DevOps according to claim 1, characterized in that: The process in which the test result tracking module participates in the automated operation and maintenance system is the test result improvement process: data test stakeholders track the test results, adjust the data to be released according to the test results, and improve the metadata standards and test case standards.

6. The DataOps platform for implementing big data construction based on DevOps according to claim 1, characterized in that: In the data service API, the steps for detecting data to be published are as follows: Step S1: Data structure detection: the data service API detects the data structure of the data to be published according to the data structure specification of big data; Step S2: Metadata standard compliance detection: the data service API detects the data to be published according to the metadata standard; Step S3: Business standard use case compliance detection: the data service API detects the data to be published according to the business standard use case; Step S4: Send the data to be published. The data that passes the metadata standard compliance and business standard use case compliance is published to the target data warehouse, and the data that fails the compliance is blocked.

Images