Method and system for generating FlinkSQL field consanguinity

A blood relationship and field technology, which is applied in the field of generating FlinkSQL field blood relationship, can solve the problem that metadata cannot generate FlinkSQL field blood relationship, etc., and achieve the effect of convenient influence range analysis and automatic pull-up

Pending Publication Date: 2022-03-25
SICHUAN XW BANK CO LTD
View PDF0 Cites 2 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0005] The technical problem to be solved by the present invention is: at present, the management method for metadata generated based on Flink SQL cannot generate Flink SQL field lineage, and the purpose is to provide a method and system for generating FlinkSQL field lineage, by customizing Flink SQL DDL and DML parsing process, parse the SQL statement to obtain the library name, table name and field name, generate an array containing all tables in SQL, then traverse the array in sequence to get the relationship between fields, and finally obtain the Flink SQL field by displaying the relationship between fields blood relationship

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Method and system for generating FlinkSQL field consanguinity
  • Method and system for generating FlinkSQL field consanguinity
  • Method and system for generating FlinkSQL field consanguinity

Examples

Experimental program
Comparison scheme
Effect test

Embodiment

[0067] At present, the management method for the metadata generated based on Flink SQL cannot generate the lineage of the Flink SQL field. For this, this embodiment provides a method for generating the lineage of the FlinkSQL field, including the following steps:

[0068] Step 1: Obtain the complete SQL statement, parse the SQL engine through Apache calcite, customize the DDL and DML parsing process for FlinkSQL, and obtain the library name, table name, and field name.

[0069] Step 1.1: Define the FlinkColumn object to represent the column name parsed in Flink SQL, including the column name (name), column data type (type) and column comment (comment); define the FlinkTable type to represent the SQL statement The table used in contains two data types: linked list array List>>, indicating the field column name mapping list of the table and the filter selection field array Map, indicating the filter selection field of the table.

[0070] The linked list array and filter select...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses a method and system for generating Flink SQL field consanguinity, and the method comprises the steps: firstly obtaining a complete SQL statement, and obtaining a library name, a table name and a field name through the analysis of the SQL statement and the self-definition of DDL and DML analysis processes for the Flink SQL; the method comprises the following steps of: obtaining a ListTable array containing all tables in an SQL (Structured Query Language), sequentially traversing the array according to the obtained ListTable array containing all the tables in the SQL to obtain a blood relationship between fields, and finally, displaying a final field result according to a layered field display mode to obtain the FlinkSQL field blood relationship. The invention discloses a method and a system for generating FlinkSQL field consanguinity, which can be used for conveniently carrying out influence range analysis on a table and realizing automatic pulling of task dependence and field-level desensitization for sensitive data.

Description

technical field [0001] The present invention relates to the technical field of data processing, in particular to a method and system for generating lineage of FlinkSQL fields. Background technique [0002] In big data technology, real-time data warehouse technology has been widely used in enterprises, and all major enterprise organizations have established or explored real-time data warehouses that meet their own business scenarios. With the increasing demand for real-time, Apache Flink has increasingly become the first choice for many companies to build real-time data warehouses; among them, Flink SQL is used more and more. Therefore, metadata management based on Flink SQL is becoming more and more important, especially the blood relationship at the field level, and it is clear to know the final target table field corresponding to the source table field. Through the field-level blood relationship, it is possible to conveniently analyze the impact range of the table, realiz...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): G06F16/2455G06F16/28
CPCG06F16/2455G06F16/283
Inventor 谭巍陈卫田浩兵张奎翟恩荣
Owner SICHUAN XW BANK CO LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap