Supercharge Your Innovation With Domain-Expert AI Agents!

Multi-data-source data detection method, device and equipment and readable storage medium

A data detection device and data detection technology, applied in the field of big data, can solve problems such as inability to support the configuration of multiple data sources

Inactive Publication Date: 2019-04-19
WEBANK (CHINA)
View PDF7 Cites 7 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0004] The main purpose of the present invention is to provide a multi-data source data detection method, device, equipment and readable storage medium, aiming to solve the technical problem that the existing data detection method cannot support the configuration of multiple data sources

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Multi-data-source data detection method, device and equipment and readable storage medium
  • Multi-data-source data detection method, device and equipment and readable storage medium

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0038] It should be understood that the specific embodiments described here are only used to explain the present invention, not to limit the present invention.

[0039] Such as figure 1 as shown, figure 1 It is a schematic structural diagram of the hardware operating environment involved in the solution of the embodiment of the present invention.

[0040] It should be noted, figure 1 That is, it is a structural schematic diagram of a hardware operating environment of a data detection device with multiple data sources. The data detection device with multiple data sources in the embodiment of the present invention may be a terminal device such as a PC or a portable computer.

[0041] Such as figure 1 As shown, the multi-data source data detection device may include: a processor 1001 , such as a CPU, a network interface 1004 , a user interface 1003 , a memory 1005 , and a communication bus 1002 . Wherein, the communication bus 1002 is used to realize connection and communica...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention discloses a multi-data-source data detection method, device and equipment and a readable storage medium, and the method comprises the steps: loading a data source driver corresponding todata sources according to a detection instruction after the detection instruction for detecting data corresponding to at least two data sources is detected; Reading to-be-detected data correspondingto the detection instruction through the data source driver, storing the to-be-detected data in a Spark cluster, and obtaining a target detection rule corresponding to the to-be-detected data; And through the target detection rule, detecting the to-be-detected data in the Spark cluster to obtain a detection result of the to-be-detected data. According to the method, cross-data-source data detection is supported through the Spark cluster, and configuration of multiple data sources is supported in the data detection process.

Description

technical field [0001] The present invention relates to the field of big data technology, in particular to a multi-data source data detection method, device, equipment and readable storage medium. Background technique [0002] Among many data processing applications, data detection is the most important link in the big data processing business. Existing data detection mainly includes Apache Griffin and Ali DataWorks Data Quality (DQC), among them, Griffin is an open source data detection solution applied to distributed data systems, such as in distributed systems such as Hadoop, Spark and Storm, Griffin provides a unified process to define and test the quality of data sets and report problems in a timely manner. Griffin has been deployed on eBay to provide services for the core data system, providing a set of common functions to solve the pain points in data quality detection. To detect data quality problems, it is mainly divided into the following steps: 1. User registrat...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
IPC IPC(8): G06F16/215G06F16/25
Inventor 陈华佳叶家豪邸帅卢道和
Owner WEBANK (CHINA)
Features
  • R&D
  • Intellectual Property
  • Life Sciences
  • Materials
  • Tech Scout
Why Patsnap Eureka
  • Unparalleled Data Quality
  • Higher Quality Content
  • 60% Fewer Hallucinations
Social media
Patsnap Eureka Blog
Learn More

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic, Popular Technical Reports.

© 2025 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap|About US| Contact US: help@patsnap.com