Data insertion method and device, equipment and storage medium

A data insertion and data technology, applied in database indexing, structured data retrieval, special data processing applications, etc., can solve problems such as system instability, excessive memory usage, and the number of open file handles, etc., to improve performance, The effect of enhancing the effect

Active Publication Date: 2019-10-22
TRANSWARP INFORMATION TECH SHANGHAI
View PDF5 Cites 1 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0003] Partitioning is a commonly used data organization method in databases. Most of the existing methods use a single partition to insert one by one, and the performance cannot meet the requirements when processing large batches of data.
If you want to insert different partitions at the same time, the system will be unstable because of operating too many files in different partitions at the same time for a long time, such as excessive memory usage, large number of open file handles, etc.

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Data insertion method and device, equipment and storage medium
  • Data insertion method and device, equipment and storage medium
  • Data insertion method and device, equipment and storage medium

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0032] figure 1 It is a flowchart of a data insertion method provided by Embodiment 1 of the present invention. This embodiment is applicable to the situation of importing data into a database. This method can be executed by a data insertion device, which can be implemented by hardware and / or software. Implementation, specifically includes the following steps:

[0033] Step 110, according to the acquired data insertion command, determine the insertion action type of the data insertion command.

[0034] Wherein, when a data insertion task needs to be performed, a corresponding insertion command will be executed, and it can be determined whether it is a dynamic partition insertion or a static partition insertion according to the data insertion command of the database engine. Static partition insertion refers to specifying the target partition when inserting data, and only one partition can be inserted at a time; dynamic partition insertion refers to not specifying the target pa...

Embodiment 2

[0046] figure 2 It is a flowchart of a data insertion method provided by Embodiment 2 of the present invention. The technical solution of this embodiment is further refined on the basis of the above technical solution, and specifically includes the following steps:

[0047] Step 210, according to the acquired data insertion command, determine the insertion action type of the data insertion command.

[0048] Step 220, when the insert action type is dynamic partition insert, determine the partition and bucket information of the target table according to the meta information of the target table.

[0049] Step 230, when the target table is a partitioned and bucketed table, group the data to be inserted according to the hash value of the column corresponding to the bucketed column.

[0050] Among them, after obtaining the partition and bucket information, if the target table is a partition and bucket table, first divide the data of the source table into bucket arrays according to t...

Embodiment 3

[0055] image 3 It is a flowchart of a data insertion method provided by Embodiment 3 of the present invention. The technical solution of this embodiment is further refined on the basis of the above technical solution, and specifically includes the following steps:

[0056] Step 310, according to the acquired data insertion command, determine the insertion action type of the data insertion command.

[0057] Step 320, when the insert action type is dynamic partition insert, determine the partition and bucket information of the target table according to the meta information of the target table.

[0058] Step 330, divide the data to be inserted into at least one group according to the partition and bucket information.

[0059] Step 340, when the target table is a multi-level partition table, sort the data to be inserted in the group according to the order of the partitions of the target table.

[0060] Wherein, if the target table is determined to be a multi-level partition tab...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The embodiment of the invention discloses a data insertion method and device, equipment and a storage medium, and the method comprises the steps: determining an insertion action type of a data insertion command according to the obtained data insertion command; when the insertion action type is dynamic partition insertion, determining partition bucket information of a target table according to meta-information of the target table; dividing to-be-inserted data into at least one group according to the partition and bucket division information; sorting the to-be-inserted data in the group according to the partition and bucket information; and according to the sequence of the to-be-inserted data in the group, dynamically inserting each group of to-be-inserted data into the corresponding targettable file in sequence. According to the technical scheme of the embodiment of the invention, the performance of dynamic partition insertion is improved under the condition of ensuring the stability of the system.

Description

technical field [0001] Embodiments of the present invention relate to data storage technologies, and in particular, to a data insertion method, device, equipment, and storage medium. Background technique [0002] With the complexity of application scenarios, data often flows between different databases. With the advent of the era of big data, the amount of data imported or exported between databases is also increasing. [0003] Partition is a commonly used data organization method in databases. Most of the existing methods use a single partition to insert one by one, and the performance cannot meet the requirements when processing large batches of data. If you want to insert different partitions at the same time, the system will be unstable due to the simultaneous operation of too many files in different partitions for a long time, such as excessive memory usage and the number of open file handles. Contents of the invention [0004] Embodiments of the present invention p...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): G06F16/22
CPCG06F16/2255G06F16/2282
Inventor 张泓毅陈振强
Owner TRANSWARP INFORMATION TECH SHANGHAI
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap