Looking for breakthrough ideas for innovation challenges? Try Patsnap Eureka!

Validating code of an extract, transform and load (ETL) tool

a technology of extracting transform and loading, applied in the field of validating etl code, can solve the problems of software developers, low quality coding, large number of jobs and other objects, etc., and achieve the effect of facilitating high quality, less time, and effective review of cod

Inactive Publication Date: 2016-01-21
MAPLEBEAR INC
View PDF0 Cites 28 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Benefits of technology

The present invention provides a tool called "Validation Script" that can automatically check and validate Extract Transform Load (ETL) code based on certain standards and provide detailed reports on any violations. This tool helps to quickly review and fix any issues with the code, reducing costs. The reports generated by the tool are easy to understand, even for non-technical managers, and can help identify areas of the code needing further analysis. The invention also includes both automatic and partial manual review processes for ETL code, with the automatic review providing a savings of at least 100% and the partial manual review providing a savings of at least 50% on review efforts.

Problems solved by technology

ETL projects involve complex logic and a large number of jobs and other objects.
Software developers drawing from different experiences may generate undisciplined coding of low quality.
Low quality code has defects and experiences malfunctions, which leads to increased cost and time requirements to rework the code to fix the defects.
In the long run, low quality code leads to a lower client confidence in the developers' ability to deliver quality code, which impacts the reputation of the developers.
Low quality code still results when developers do not comply with the standards.
Not complying with the standards may result from various reasons, including the lack of experience of a developer, the developer being unaware of the relevance of a particular standard and its impact, and / or overlooking a standard by mistake.
Overlooking a standard is the most common source of non-compliance with the standards because it is easy for a developer to miss one of the hundreds of standards that may be set for a particular project.
Manual reviews of hundreds of standards are time consuming processes and can be difficult to execute in ETL projects with aggressive timelines.
Even if manual reviews are done, reviewers can make mistakes similar to the developers by overlooking some of the standards.
With only manual reviews in place, it is difficult to assure a 100% adherence to standards and the quality of deliverables is always at risk.

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Validating code of an extract, transform and load (ETL) tool
  • Validating code of an extract, transform and load (ETL) tool
  • Validating code of an extract, transform and load (ETL) tool

Examples

Experimental program
Comparison scheme
Effect test

examples

[0060]FIG. 3 depicts an exemplary portion 300 of validation report 110 (see FIG. 1) summarizing violations of standards determined by the process of FIG. 2, where the report is generated by the system of FIG. 1, in accordance with embodiments of the present invention. Portion 300 includes a section 302 including a list of parallel jobs and a section 304 including a list of sequence jobs, which are determined by validation tool 104 (see FIG. 1).

[0061]Section 302 includes object-specific information for each job, including a job name, aggregator count, transformer count, re-partitioning count, sort count, annotation count, database read / write count, file read / write count, hardcoding count (i.e., count of the number of instances of hardcoding for each job), and standards violation count.

[0062]For example, for the second row of data in section 302, the job name is AIE001WriteSAPFile, the aggregator count is 8, the transformer count is 6, the re-partitioning count is 6, the sort count is...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

An approach for validating code for an extract, transform and load (ETL) tool is provided. Naming, coding, and performance standards for the code is received. The code is exported to a job definition file and parsed. Violations of the standards are determined by determining the parsed code does not match the standards. A report identifying the violations is generated. Based on a review of the report and a rework of the code to comply with the standards, the reworked code is exported to another job definition file and parsed, the parsed reworked code is determined to not include the violations of the standards, and a second report is generated that indicates that the reworked code does not include the violations. An approval of the reworked code is received based on the second report.

Description

TECHNICAL FIELD[0001]The present invention relates to reviewing ETL code, and more particularly to validating ETL code to accelerate review of ETL projects.BACKGROUND[0002]ETL projects involve complex logic and a large number of jobs and other objects. For example, each project may include 100 to 1000 jobs and other objects. Effective and quality delivery demands disciplined coding. Software developers drawing from different experiences may generate undisciplined coding of low quality. Low quality code has defects and experiences malfunctions, which leads to increased cost and time requirements to rework the code to fix the defects. In the long run, low quality code leads to a lower client confidence in the developers' ability to deliver quality code, which impacts the reputation of the developers. For any project, code quality standards are set before development activities start. The standards specify how to name objects, the best coding practices to be followed, etc. Low quality ...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
IPC IPC(8): G06F11/36
CPCG06F11/3604G06F9/44589G06F16/254G06F16/2365G06F16/2379G06F11/3612G06F11/3616
Inventor VILAKKUMADATHIL, ROKKY
Owner MAPLEBEAR INC
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Patsnap Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Patsnap Eureka Blog
Learn More
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic, Popular Technical Reports.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap|About US| Contact US: help@patsnap.com