Method and device for monitoring data quality

A technology for monitoring data and quality, applied in the computer field, can solve problems such as untimely feedback of data error information, failure to detect data errors, waste of database resources and time, etc., to reduce the possibility of errors, ensure data quality, and avoid repeated acquisitions Effect

Inactive Publication Date: 2019-07-16
BEIJING JINGDONG SHANGKE INFORMATION TECH CO LTD +1
View PDF5 Cites 5 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0005] 1) Since the data calculation task includes data verification, the source code of the task is complex and the amount of data development is large;
[0006] 2) It is inconvenient to modify the data verification strategy: After the data verification program is written, it is very inconvenient to modify the data verification strategy; since the source code of the data calculation task needs to be directly modified, the risk of source code errors will also increase;
[0007] 3) The feedback of data error information is not timely: the source data can only be verified when the data calculation task is executed, and the data error cannot be found before the data calculation task is executed;
[0008] 4) There is repeated verification: every time a data calculation task is performed, the data needs to be verified, which wastes database resources and time

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Method and device for monitoring data quality
  • Method and device for monitoring data quality
  • Method and device for monitoring data quality

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0052] Exemplary embodiments of the present invention are described below in conjunction with the accompanying drawings, which include various details of the embodiments of the present invention to facilitate understanding, and they should be regarded as exemplary only. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the invention. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.

[0053] figure 1 is a schematic flow diagram of the main flow of a method for monitoring data quality according to an embodiment of the present invention, such as figure 1 As shown, methods for monitoring data quality include:

[0054] Step S101, using the source data and / or output data of the data calculation task as the data to be verified;

[0055] Step S102, acquirin...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses a method and device for monitoring data quality, and relates to the technical field of computers. A specific embodiment of the method comprises the following steps: taking source data and / or output data of a data calculation task as to-be-verified data; obtaining a verification rule of the to-be-verified data from a preset data verification rule set; executing a data verification task on the to-be-verified data based on the verification rule, wherein the data verification task and the data calculation task are carried out asynchronously. According to the embodiment of the invention, the data quality can be ensured, and the possibility of data calculation task errors is greatly reduced; the workload and complexity of data calculation and verification are reduced; data verification rules can be modified conveniently, and personalized customization of a data verification strategy is supported.

Description

technical field [0001] The invention relates to the field of computer technology, in particular to a method and device for monitoring data quality. Background technique [0002] Different data marts need to share data, and different data calculation tasks (including data model calculation, data analysis calculation, etc.) generate new data by processing old data. Data errors may be caused by database node loss (unavailable), node crash, upstream data error, etc. If you can't detect data errors in advance and continue to let the erroneous data flow downstream, it will have an impact on downstream related businesses, and may even have a fatal impact on some key businesses. If the data is found to be wrong after the data has been affected, it may cause huge losses, and it will be more difficult to remedy the wrong data. [0003] In order to avoid the occurrence of the above situation, in the prior art, the data is verified when the data calculation task is performed, that is,...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): G06F16/215G06F16/23G06F11/32
CPCG06F11/327G06F16/215G06F16/2365
Inventor 李立威罗翔游正朋
Owner BEIJING JINGDONG SHANGKE INFORMATION TECH CO LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products