Data association query method and system in RocksDB-based key-value storage mode
What is Al technical title?
Al technical title is built by PatSnap Al team. It summarizes the technical point description of the patent document.
A technology of data association and storage mode, applied in the field of operator optimization, to avoid overhead, improve computing efficiency, and improve efficiency
Pending Publication Date: 2020-10-09
SHANDONG LANGCHAO YUNTOU INFORMATION TECH CO LTD
View PDF0 Cites 3 Cited by
Summary
Abstract
Description
Claims
Application Information
AI Technical Summary
This helps you quickly interpret patents by identifying the three key elements:
Problems solved by technology
Method used
Benefits of technology
Problems solved by technology
[0005] The technical task of the present invention is to provide a data association query method and system based on the key-value storage mode of RocksDB, to solve
Method used
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more
Image
Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
Click on the blue label to locate the original text in one second.
Reading with bidirectional positioning of images and text.
Smart Image
Examples
Experimental program
Comparison scheme
Effect test
Embodiment 1
[0054] as attached image 3 As shown, the data association query method based on the key-value storage mode of RocksDB of the present invention, the method is to store each line of data through the channel of the go language under the key-value storage system and then read it piece by piece. Realize block reading, that is, process data in units of blocks. After receiving the data block, distribute the data to different Join Workers for parallel computing to improve the computing performance of the Hash Join operator; the details are as follows:
[0055] S1. Read data block by block instead of line by line; the details are as follows:
[0056] S101. Store the read data into the channel without performing calculations;
[0057] S102. When the amount of data in the channel satisfies the shipping conditions, divide the channel into blocks for calculation.
[0058] S2. Each module communicates through the channel cache, so that each module is executed asynchronously, and the exec...
Embodiment 2
[0077] The present invention is based on the data correlation query system under the key-value storage mode of RocksDB, and this system comprises,
[0078] The main thread is used to read all the small table data and construct a hash table; then start the Table Reader and JoinWorker to start background work, and finally return the Join result calculated by the Join Worker;
[0079] Table Reader thread, used to read the data of the Outer table and distribute it to each Join Worker;
[0080] Multiple Join Worker operators are used to look up the data in the hash table, Join matching Inner and Outer tables, and pass the results to the Main Thread for parallel computing.
Embodiment 3
[0082] The embodiment of the present invention also provides an electronic device, including: a memory and a processor;
[0083] Wherein, the memory stores computer-executable instructions;
[0084] The one processor executes the computer-executed instructions stored in the memory, so that the one processor executes the data association query method in the key-value storage mode of RocksDB as in Embodiment 1.
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more
PUM
Login to view more
Abstract
The invention discloses a data association query method and system in a RocksDB-based key-value storage mode and belongs to the technical field of operator optimization. The technical problem to be solved by the invention is how to improve the efficiency of traditional hash join calculation based on line-by-line reading and serial operation. The associated query efficiency is improved, the adoptedtechnical scheme is as follows: the method comprises the following steps: ; the method is applied to a key-value storage system. Each row of data is stored through a channel of go language, and thenthe data is read one by one to realize block reading, i.e., the data is processed by taking a block as a unit, and after the data block is received, the data is distributed to different Join Workers for parallel computing, so that the computing performance of a Hash Join operator is improved; the method specifically comprises the following steps: S1, reading data block by block; s2, all the modules are in cache communication through channels, all the modules are made to be executed asynchronously, and all the modules are made to end execution by sending end marks; and S3, calculating the mutual concurrent execution of the Join Workers.
Description
technical field [0001] The invention relates to the technical field of operator optimization, in particular to a data association query method and system based on a RocksDB-based key-value storage mode. Background technique [0002] RocksDB uses a log-structured database engine, which is written in C++ for better performance. Key and value are byte streams of arbitrary size. RocksDB is specially optimized for fast and low-latency storage devices (such as flash memory or high-speed hard disk). RocksDB will maximize the high rate read and write performance of flash memory and RAM. RocksDB is suitable for many different workload types. From data storage engines like MyRocks, to application data caches, and even some embedded workloads, RocksDB can calmly face these different data workload requirements. RocksDB provides some basic operations, such as opening and closing the database. Read and write support is also provided for advanced operations such as merge and compressi...
Claims
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more
Application Information
Patent Timeline
Application Date:The date an application was filed.
Publication Date:The date a patent or application was officially published.
First Publication Date:The earliest publication date of a patent with the same application number.
Issue Date:Publication date of the patent grant document.
PCT Entry Date:The Entry date of PCT National Phase.
Estimated Expiry Date:The statutory expiry date of a patent right according to the Patent Law, and it is the longest term of protection that the patent right can achieve without the termination of the patent right due to other reasons(Term extension factor has been taken into account ).
Invalid Date:Actual expiry date is based on effective date or publication date of legal transaction data of invalid patent.
Login to view more
IPC IPC(8): G06F16/2453G06F16/2455
CPCG06F16/24532G06F16/24552
Inventor 牟冠学史大义柴毅
Owner SHANDONG LANGCHAO YUNTOU INFORMATION TECH CO LTD
Login to view more 
Login to view more