Hierarchical storage method and device for full-text retrieval
A hierarchical storage and full-text technology, which is applied in the field of hierarchical storage methods and devices for full-text retrieval, can solve the problems of low retrieval frequency of Class B services, low retrieval frequency of old data, and low resource utilization, so as to optimize retrieval performance and improve The effect of comprehensive query speed
Inactive Publication Date: 2021-11-02
NAT COMP NETWORK & INFORMATION SECURITY MANAGEMENT CENT
View PDF6 Cites 0 Cited by
- Summary
- Abstract
- Description
- Claims
- Application Information
AI Technical Summary
Problems solved by technology
[0004] 1) The recent data retrieval frequency is high, while the old data retrieval frequency is low;
[0005] 2) The retrieval frequency of Class A business data is high, while the retrieval frequency of Class B business is low;
This brings new problems such as low resource utilization, resource waste, additional manual maintenance, and inconvenient use.
Method used
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View moreImage
Smart Image Click on the blue labels to locate them in the text.
Smart ImageViewing Examples
Examples
Experimental program
Comparison scheme
Effect test
Embodiment Construction
[0050]Embodiments of the present invention provide a method and specific implementation of disk hierarchical storage for full-text retrieval scenarios. It includes the format definition and metadata storage of hierarchical storage strategy, the automatic calculation partition of loading engine and the method of mapping according to hierarchical strategy, and the control and implementation method of hierarchical storage. The so-called hierarchical storage refers to the use of different performance storage resources such as SATA disks and SSD disks for different data in the same cluster. For a large cluster, we can formulate storage strategies according to the retrieval frequency and performance requirements of different data, store them on different disks, and realize automatic management and automatic migration at the same time, so that the cluster resources can be used reasonably and have a wide range of application scenarios. With the continuous improvement of the level of i...
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More PUM
Login to View More Abstract
The invention discloses a hierarchical storage method and device for full-text retrieval. The method includes: obtaining SQL statements, creating a full-text retrieval table, and persisting the full-text retrieval table into Zookeeper; configuring a part of nodes in the ElasticSearch cluster to use SSD disk, another part of the nodes use SATA disk, and install a custom ElasticSearch plug-in on each node; the data loading tool loads the document data into the ElasticSearch cluster through the call interface API of the ElasticSearch cluster, and executes the request through the ElasticSearch plug-in Filter, and use the metadata in the pre-stored full-text retrieval table to create an index; run the monitoring strategy through the ElasticSearch plug-in, monitor the changes in the metadata information of the table in Zookeeper, apply the hierarchical storage strategy, and execute the timing rollback strategy, Send the rollback task to the ElasticSearch cluster.
Description
technical field [0001] The invention relates to the field of big data processing NOSQL, in particular to a hierarchical storage method and device for full-text retrieval. Background technique [0002] With the continuous development of Internet technology and the continuous improvement of informatization, the amount of data has grown rapidly, and the storage and application of supporting massive data have also flourished. Among them, in the field of document retrieval, the open source project Elasticsearch has gained wide attention and application. Elasticsearch is an open source, highly scalable, distributed full-text search engine that can store and retrieve data in near real-time. It has good scalability and can be extended to hundreds of servers to process PB-level data. In the Internet and enterprise applications, inverted search has a wide range of applications, such as log monitoring, web search, hot search, and entity feature label retrieval. These requirements corr...
Claims
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More Application Information
Patent Timeline
Login to View More Patent Type & Authority Patents(China)
IPC IPC(8): G06F16/31G06F16/33
Inventor 刘欣然张鸿惠榛吕雁飞马秉楠李斌斌王振宇黄航王树鹏
Owner NAT COMP NETWORK & INFORMATION SECURITY MANAGEMENT CENT