ETL methods, apparatus, devices and media based on data pipelines and security sandboxes

A data pipeline and security sandbox technology, applied in the computer field, can solve problems such as large performance loss, low ETL transmission speed, and packet conflict, and achieve the effect of improving collection efficiency

Active Publication Date: 2022-06-07
FUJIAN NEWLAND SOFTWARE ENGINEERING CO LTD
View PDF5 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0010] The technical problem to be solved by the present invention is to provide an ETL method, device, device, and medium based on data pipelines and security sandboxes, thereby solving the problems of low ETL transmission speed, large performance loss, strong correlation between components and services, and parental problems. The delegation model has a package conflict defect problem in the traditional ETL collection scheme

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • ETL methods, apparatus, devices and media based on data pipelines and security sandboxes
  • ETL methods, apparatus, devices and media based on data pipelines and security sandboxes
  • ETL methods, apparatus, devices and media based on data pipelines and security sandboxes

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0042] like Figure 5 As shown, this embodiment provides an ETL method based on a data pipeline and a security sandbox, including the following steps:

[0043] S1. Start the actuator through the startup module;

[0044] S2. The executor receives the task message information of the front end, extracts the component codes corresponding to the extraction, conversion and loading according to the task message information, and instantiates the extraction component, the conversion component and the loading component respectively according to the component codes, and A data pipeline between the extraction component, the conversion component and the loading component is constructed through a lock-free memory queue, thereby constructing an ETL topology instance; wherein,

[0045] The extraction component, the conversion component and the loading component are formed by decoupling a single component that realizes the ETL function, are independent and reusable components, and can be free...

Embodiment 2

[0055] like Figure 7 As shown, in this embodiment, an ETL device based on a data pipeline and a security sandbox is provided, including:

[0056] The startup module is used to start the executor, specifically using the executor ClassLoader to load the executor;

[0057] The executor is used for receiving the task message information of the front end, according to the task message information, extracts the component codes corresponding to extraction, conversion and loading, and instantiates the extraction component, the conversion component and the loading component respectively according to the component codes; And build the data pipeline between the extraction component, the conversion component and the loading component through a lock-free memory queue, thereby constructing an ETL topology instance; wherein, the extraction component, the conversion component and the loading component It is decoupled from a single component that implements the ETL function, is an independen...

Embodiment 3

[0067] This embodiment provides an electronic device, such as Figure 8 As shown, a memory, a processor, and a computer program stored in the memory and running on the processor are included. When the processor executes the computer program, any implementation manner of the first embodiment can be implemented.

[0068] Since the electronic device introduced in this embodiment is the device used to implement the method in the first embodiment of the present application, based on the method introduced in the first embodiment of the present application, those skilled in the art can understand the electronic device in this embodiment. The specific implementation manner and various modifications thereof, so how the electronic device implements the methods in the embodiments of the present application will not be described in detail here. As long as the devices used by those skilled in the art to implement the methods in the embodiments of the present application fall within the sco...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The present invention provides an ETL method, device, device and medium based on a data pipeline and a security sandbox. The method includes: starting the executor to receive the front-end task message information, extracting the component codes of each stage and instantiating each stage respectively Components use lock-free memory queues to build data pipelines between components at each stage, thereby constructing ETL topology instances, starting data collection, and data transmission through data pipelines, so that intermediate data does not fall to the ground, and data is produced and consumed at the same time; among them, each Stage components are independent and reusable components, and can be freely connected according to business scenarios; the data structure is an array that has been encapsulated and transformed; the class loading is performed using an extended parental delegation model, so that each stage component of the executor Run in their own independent security sandbox. In this way, the problems of low transmission speed, large performance loss, strong correlation between components and business, and package conflict defects in the parental delegation model in the current ETL collection solution are solved.

Description

technical field [0001] The present invention relates to the field of computer technology, and in particular, to an ETL method, apparatus, device and medium. Background technique [0002] ETL, short for Extract-Transform-Load in English, is used to describe the process of extracting, transforming, and loading data from the source to the destination. [0003] A component is a unit of assembly that has a convention-specific interface and a well-defined dependency environment. In this case, the component is used to realize the ETL function of a specific business, which is a general term for a software function package. [0004] The ETL acquisition device is divided into three stages, namely extraction, conversion, and loading. The data source of each stage depends on the data results processed by the previous module, that is, the data in the transformation stage depends on the output data in the extraction stage; the data in the loading stage depends on the output data in the ...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Patents(China)
IPC IPC(8): G06F16/25G06F9/54G06F21/53
Inventor 梁长生潘桦陈凯杰郑涛
Owner FUJIAN NEWLAND SOFTWARE ENGINEERING CO LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap