Heterogeneous data source synchronization quality monitoring method and device and electronic equipment
A technology of heterogeneous data sources and synchronization quality, applied in the field of big data processing, can solve problems such as waste of resources and failure to process in real time, achieve clear responsibilities, improve maintenance capabilities and execution efficiency, and improve effective utilization.
Inactive Publication Date: 2022-02-18
GUANGZHOU XUANWU WIRELESS TECH CO LTD
View PDF1 Cites 1 Cited by
- Summary
- Abstract
- Description
- Claims
- Application Information
AI Technical Summary
Problems solved by technology
Due to the limitations of the computing framework, for the chain task process, if the data quality does not meet the standard, the data problem cannot be processed in real time, and the task cannot be interrupted. Secondary processing is required based on the quality report, which is inevitable cause waste of resources
Method used
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View moreImage
Smart Image Click on the blue labels to locate them in the text.
Smart ImageViewing Examples
Examples
Experimental program
Comparison scheme
Effect test
Embodiment Construction
[0041] In order to make the purpose, technical solution and advantages of the present application clearer, the embodiments of the present application will be further described in detail below in conjunction with the accompanying drawings.
[0042] It should be clear that the described embodiments are only some of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the embodiments of the present application, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts belong to the protection scope of the embodiments of the present application.
[0043] The terms used in the embodiments of the present application are only for the purpose of describing specific embodiments, and are not intended to limit the embodiments of the present application. The singular forms "a", "said" and "the" used in the embodiments of this application and the appended claims are also intended to include ...
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More PUM
Login to View More Abstract
The embodiment of the invention relates to a heterogeneous data source synchronization quality monitoring method and device. The heterogeneous data source synchronization quality monitoring method provided by the embodiment of the invention comprises the following steps: collecting a pre-configured monitoring rule from a scheduling platform; according to a preset rule protocol, converting the monitoring rule into configuration information, and transmitting the configuration information to a calculation framework; the calculation framework analyzes the configuration information through a rule protocol to obtain a monitoring rule; the calculation framework uses a monitoring rule for each line of produced data, judges whether each line of data meets the monitoring rule or not, and performs statistical counting on a judgment result to obtain data quality monitoring data; and when the data output is finished, outputting the data quality monitoring data in a counter mode. According to the heterogeneous data source synchronization quality monitoring method provided by the embodiment of the invention, diversified monitoring rules can be configured in advance, and the diversity of monitoring data output is realized through a calculation framework.
Description
technical field [0001] The embodiments of the present application relate to the technical field of big data processing, in particular to a method and device for monitoring the synchronization quality of heterogeneous data sources. Background technique [0002] The existing data quality monitoring solution is aimed at MapReduce jobs and Hive jobs. According to business requirements, configure corresponding data quality rules, use the calculation framework to judge the rules during the data output process, and finally pass the output results of the data through the counter Generate quality reports. The existing data quality monitoring technology is built under the framework of big data computing, rationally utilizes cluster resources, monitors data quality while producing data under the computing framework, and generates detailed quality reports for the output results of MapReduce jobs and Hive jobs. [0003] However, this solution relies on clusters and big data computing fr...
Claims
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More Application Information
Patent Timeline
Login to View More IPC IPC(8): G06F11/30G06F16/215
CPCG06F11/3051G06F16/215
Inventor 叶长航林沛欣黄应棣陈永辉李海荣
Owner GUANGZHOU XUANWU WIRELESS TECH CO LTD