Data processing method and system, terminal equipment and storage medium

A data and data element technology, applied in the fields of terminal equipment and storage media, systems, and data governance methods, can solve the problems of waste of time and resources, unfavorable data unified verification, error reporting, etc., to reduce workload and labor costs, improve Effectiveness of Data Governance, Increased Efficiency and Reliability

Pending Publication Date: 2021-02-23
广州光点信息科技股份有限公司
View PDF0 Cites 12 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0004] However, in the process of research and practice of the prior art, the inventors of the present invention found that the prior art not only increased the developer's workload and manpower due to the need for relevant developers to manually develop corresponding data governance scripts. Cost, data governance is inefficient, and it is prone to error reporting; in the process of data governance, the existing data governance method is generally a single mapping (1-to-1 mapping) between the original data and the target data, but in the face of many-to-many When mapping, the workload of mapping and rule conversion will increase significantly according to the number of original tables and data tables, resulting in a huge waste of time and resources; at the same time, existing technologies are verifying or verifying data based on data governance results. When adjusting, there are many quality rules and different standards, which is not conducive to the unified verification of data, and the efficiency is low

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Data processing method and system, terminal equipment and storage medium
  • Data processing method and system, terminal equipment and storage medium
  • Data processing method and system, terminal equipment and storage medium

Examples

Experimental program
Comparison scheme
Effect test

no. 1 example

[0046] see Figure 1-4 .

[0047] Such as figure 1 As shown, this embodiment provides a data governance method, at least including the following steps:

[0048] S1. Connect several heterogeneous data sources through the data adapter interface of the data center, and collect the business data to be processed.

[0049] Specifically, for step S1, firstly, through the data adaptation interface of the data center together with various heterogeneous data sources, the corresponding business data to be processed is collected according to the business scenario. Heterogeneous data sources include relational databases, big data platforms, MPP databases, file databases, NO SQL databases, multidimensional databases and service databases, etc.

[0050] In a specific embodiment, the step S1 provides a data connection function, and the system supports visually configuring data connections in a browser, supports mainstream relational databases and big data platforms; Automatically identify...

no. 2 example

[0086] see Figure 5-6 .

[0087] Such as Figure 5 As shown, this embodiment provides a data governance system, including:

[0088] The data collection module 100 is used to connect several heterogeneous data sources through the data adapter interface of the data center, and collect business data to be processed.

[0089] Specifically, for the data collection module 100, firstly, through the data adaptation interface of the data center platform together with various heterogeneous data sources, corresponding business data to be processed is collected according to business scenarios. Heterogeneous data sources include relational databases, big data platforms, MPP databases, file databases, NO SQL databases, multidimensional databases and service databases, etc.

[0090] In a specific embodiment, the data collection module 100 also provides a data connection function, the system supports visually configuring data connections in a browser, supports mainstream relational databa...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses a data processing method and system, terminal equipment and a storage medium, and the method comprises the steps: enabling a plurality of heterogeneous data sources to be communicated through a data adaption interface of a data center station, and collecting to-be-processed business data; constructing and generating a corresponding data model according to the to-be-processed business data; after basic information is defined in the data center station through a data standard preset by a user, establishing corresponding data elements and data element standard rules according to service conditions; binding metadata of all field levels in the data model with data standard meta rules one by one; and performing matching verification on all metadata in the data model according to the bound data element standard rule to obtain a corresponding quality evaluation result. By defining the quality rules and the mapping quality rules, data governance can be automatically completed in batches, the efficiency and reliability of data governance are improved, the workload and labor cost of developers are reduced, and time and resource waste is avoided.

Description

technical field [0001] The present invention relates to the technical field of data processing, in particular to a data management method, system, terminal equipment and storage medium. Background technique [0002] With the rapid development of big data technology, more and more companies are beginning to pay attention to their own data problems, and start to use certain methods in enterprise data management and data planning to control data, realize big data governance, obtain governance results, and relevant users Relevant data optimization and adjustment operations can be performed according to the governance results. [0003] The steps of the existing data governance method based on structured data mainly include inputting raw data, governing the raw data, merging the governed data, and outputting the result data. Data governance scripts are manually developed by relevant data governance developers according to governance rules, and then the data is governed through th...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): G06F16/2457G06F16/2455G06F16/27G06F16/25G06F16/22G06F16/215
CPCG06F16/215G06F16/2291G06F16/24564G06F16/24573G06F16/254G06F16/27
Inventor 许可
Owner 广州光点信息科技股份有限公司
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap