Method, device and computer readable medium for data loading

A data loading and data technology, applied in the field of database information processing, can solve problems such as slow extraction efficiency, large memory usage, and unreasonable memory allocation

Active Publication Date: 2021-04-23
HUNDSUN TECH
View PDF3 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

However, when files are imported into the database and these tools are considered from the perspective of memory allocation, for some files, especially large files, once the file data is obtained, the existing tools will allocate fixed memory for the file for processing, so , in the current ETL data loading environment, the extraction efficiency is generally slow and the memory usage is large
Moreover, since the amount of data in each file is different for different files to be processed, there is also a situation where the memory allocation is unreasonable, and there are sometimes many memory remnants after the file data is imported into the data The problem

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Method, device and computer readable medium for data loading
  • Method, device and computer readable medium for data loading
  • Method, device and computer readable medium for data loading

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0030] Example embodiments will now be described more fully with reference to the accompanying drawings. Example embodiments may, however, be embodied in many forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the concept of example embodiments to those skilled in the art. The same reference numerals denote the same or similar parts in the drawings, and thus their repeated descriptions will be omitted.

[0031] Furthermore, the described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided in order to give a thorough understanding of embodiments of the invention. However, those skilled in the art will appreciate that the technical solutions of the present invention may be practiced without one or more of the ...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The present application discloses a method, device and computer-readable medium for data loading. Relating to the field of database information data, the method includes: importing a file to be processed into a local disk, and the file to be processed includes at least one subfile; performing preprocessing on the file to be processed to generate paged data; quickly operating the pool pair The paged data is subjected to data conversion processing to obtain converted data, wherein, during the data conversion process, memory is automatically allocated by a memory interceptor; and the converted data is sent to the target database. The method, device and computer-readable medium for data loading disclosed in the present application can optimize the memory usage, automatically allocate the memory, greatly reduce the memory usage, and scientifically and rationally allocate the memory usage required for each import.

Description

technical field [0001] The present invention relates to the field of database information processing, in particular to a method, device and computer-readable medium for data loading. Background technique [0002] Data loading (ETL, Extract-Transform-Load) is used to describe the process of extracting, transforming, and loading data from the source to the destination. In the actual situation, the user extracts the required data from the data source, after data cleaning, finally imports the data into the data warehouse according to the pre-defined data warehouse model. The whole process is the data loading process. [0003] In the prior art, there are many mature data processing and conversion tools. Compared with the mature conversion tools on the market, most of the tools are already perfect and powerful, and can support the import of various databases. However, when files are imported into the database and these tools are considered from the perspective of memory allocatio...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Patents(China)
IPC IPC(8): G06F16/25
Inventor 朱少伟吴成英
Owner HUNDSUN TECH
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap