Looking for breakthrough ideas for innovation challenges? Try Patsnap Eureka!

A data query engine based on shard storage and half index

A data query engine and data query technology, applied in the field of data query engines, can solve the problems of high query speed and decrease in overall efficiency, and achieve the effect of improving efficiency, fast query speed, and fast positioning query speed.

Active Publication Date: 2020-12-18
CHINA RAILWAY ENGINEERING EQUIPMENT GROUP CO LTD
View PDF6 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

At the same time, the larger the amount of data, the faster the query speed will be, and the overall efficiency will drop significantly

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0033] Embodiment 1: A data query engine based on shard storage and half-index, including a data generation module, a data sorting module, and a data query module.

[0034] Wherein, the data generation module is used to generate the plaintext of each MD5 encrypted data in each file in the form of fragment storage, including an MD5 encryption unit, a file name generation unit and a folder name generation unit.

[0035] The MD5 encryption unit is used to perform MD5 encryption on plaintext data.

[0036] The file name generating unit converts the hexadecimal of the 9th to 13th digits of the 32-bit MD5 encrypted data into decimal as the file name.

[0037] The folder name generating unit divides the file name by 1024 and removes the remainder as the folder name, and the file is stored in the current folder.

[0038] The data sorting module sorts the plaintext data in each folder from small to large, including a data sorting unit, which is used to reverse the 21st to 24th bits of...

Embodiment 2

[0040] Embodiment 2: A query method of a data query engine based on shard storage and a half-index, comprising the following steps:

[0041] Step 1. Query the plaintext of a piece of 32-bit MD5 encrypted data.

[0042] Step 2. Convert the 9th to 13th digits of the MD5 value of the current plaintext into decimal to obtain the file name.

[0043] Step 3: Divide the file name by 1024 to obtain the folder name, and then obtain the storage location of the file corresponding to the current plaintext.

[0044] Step 4. Enter the storage location of the found file, open the current file, locate the middle part of the content of the current file, and start to perform a split query. Positioning can be implemented with existing algorithms, and there are many mature algorithms that can be implemented.

[0045] Step 5, perform MD5 encryption on the middle position data of the current file.

[0046] Step 6. Convert the 21st to 24th digits of the encrypted 32-bit MD5 value of the current f...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention discloses a data query engine based on fragmented storage and a folded index. The data query engine comprises a data generation module, a data sorting module and a data query module. Themethod comprises the following steps: storing various plaintext data in a specific file based on GPU operation; dynamically sorting the data in each file, and sorting the data from small to large based on 21-24 bits of each line of plaintext data MD5 values in an inverted order; and finally, querying plaintext data corresponding to the specific MD5 encrypted data based on the half index. According to the method, firstly, the database data storage pressure is effectively relieved, and only a plaintext is needed to be stored instead of a plaintext and MD5 simultaneous storage mode adopted by anoriginal storage method; meanwhile, the file generation and data storage efficiency is improved based on a fragmentation technology; based on the half-fold query algorithm, the file content rapid positioning query speed is greatly improved, and the advantage that the query speed is higher under the condition that the data volume is larger is fully shown.

Description

technical field [0001] The invention relates to a data query engine, in particular to a data query engine based on sliced ​​storage and a half index. Background technique [0002] For the traditional database query MD5 implementation method includes the following two steps: [0003] The first step: create data; [0004] 1. Create two tables in the database first, and the table names are text and MD5 respectively; [0005] 2. Put the plaintext and the MD5 generated by the plaintext into the database at the same time; [0006] 3. Create an index; [0007] The second step: query data; [0008] 1. Traverse the index first; [0009] 2. Obtain plaintext by matching MD5 with a relational database. [0010] This method stores plaintext and 32-bit ciphertext at the same time, and creates an index for easy query. Compared with the above method that only stores plaintext and calculates the index, it takes up double the space. At the same time, the larger the amount of data, the ...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & Authority Patents(China)
IPC IPC(8): G06F16/13G06F16/14H04L9/06
Inventor 段俊辉朱永超张杨杨徐受天
Owner CHINA RAILWAY ENGINEERING EQUIPMENT GROUP CO LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Patsnap Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Patsnap Eureka Blog
Learn More
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic, Popular Technical Reports.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap|About US| Contact US: help@patsnap.com