Looking for breakthrough ideas for innovation challenges? Try Patsnap Eureka!

Construction method and application of distributed learning index model

A construction method and distributed technology, applied in the field of computer distributed storage, can solve the problem of high CPU overhead of storage nodes, and achieve the effect of reducing CPU overhead

Pending Publication Date: 2021-12-10
HUAZHONG UNIV OF SCI & TECH
View PDF0 Cites 1 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0007] Aiming at the above defects or improvement needs of the prior art, the present invention provides a construction method and application of a distributed learning index model to solve the technical problem of high CPU overhead of storage nodes in the prior art

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Construction method and application of distributed learning index model
  • Construction method and application of distributed learning index model
  • Construction method and application of distributed learning index model

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0047] A method for constructing a distributed learning index model, such as figure 1 shown, including:

[0048] For each storage node, after sorting the stored data according to the size of the key value key, the key value key of the stored data is used as the input, and the corresponding sorting position is used as the output, and the machine learning model is trained to obtain the key value of each storage node Learn the index model and synchronize it to all computing nodes; the data in this embodiment includes a key value key and a pointer pointing to a value value;

[0049] The learning index model includes multiple independent index sub-models; the data stored in the storage node is divided into multiple data intervals according to the key value key, each index sub-model is used to index the data in a data interval, and each The data intervals covered by the index sub-models do not overlap each other; each index sub-model is trained by the data in the corresponding data...

Embodiment 2

[0058] An insertion method of a learning index model constructed based on the construction method of a distributed learning index model provided in Embodiment 1, such as figure 2 shown, including the following steps:

[0059] S11. The calculation node calculates the sorting position of the data to be inserted by using the learning index model on it;

[0060] Specifically, determine the corresponding index sub-model in the learning index model according to the key value key of the data to be inserted, and input the key value key of the data to be inserted into the index sub-model to obtain the sorting position of the data to be inserted;

[0061] S12. Based on the address conversion table on the computing node (the address conversion table corresponding to the index sub-model corresponding to the data interval where the data to be inserted) converts the obtained sorting position into the physical position of the corresponding array, the computing node passes the unilateral Th...

Embodiment 3

[0088] A method for querying a learning index model constructed based on the method for constructing a distributed learning index model provided in Embodiment 1, comprising the following steps:

[0089] S21. The calculation node uses the learning index model on it to calculate the sorting position of the data to be queried;

[0090] Specifically, determine the corresponding index sub-model in the learning index model according to the key value key of the data to be queried, and input the key value key of the data to be queried into the index sub-model to obtain the sorting position of the data to be queried;

[0091] S22. Based on the address conversion table on the computing node (the address conversion table corresponding to the index sub-model corresponding to the data interval where the data to be queried) converts the obtained sorting position into the physical position of the corresponding array, the computing node passes the unilateral The RDMA operation reads the corre...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention discloses a construction method and application of a distributed learning index model, and belongs to the technical field of computer distributed storage. The construction method comprises the following steps: for each storage node, sorting data stored in the storage node according to the size of key values, taking the key values of the stored data as input, taking corresponding sorting positions as output, training a machine learning model to obtain a learning index model of each storage node, and synchronizing the learning index model to all calculation nodes; the calculation node directly modifies data in the storage node through RDMA operation, and a CPU of the storage node does not need to participate in work; meanwhile, the calculation node asynchronously retrains the old model and synchronizes the new model into the storage node; the operation of modifying the data and the model is executed in the calculation node in the distributed system, so that the CPU overhead of the storage node is greatly reduced.

Description

technical field [0001] The invention belongs to the technical field of computer distributed storage, and more specifically relates to a construction method and application of a distributed learning index model. Background technique [0002] Facing the demand for massive data access, the data center stores the data in a distributed system, and connects the memories of different machines through the network to provide high-capacity, efficient data storage and access services. Dividing different machines in a distributed system into storage nodes and computing nodes can have greater flexibility and scalability. Computing nodes can directly read and write the memory of storage nodes through RDMA technology, further improving network-based distributed storage. system performance. Storage systems use different index structures to meet different requirements, among which the tree index structure is an important structure to satisfy range requests. However, the existing tree index...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & Authority Applications(China)
IPC IPC(8): G06F16/27G06F16/23G06F16/22G06K9/62
CPCG06F16/27G06F16/2228G06F16/23G06F18/214
Inventor 华宇李鹏飞
Owner HUAZHONG UNIV OF SCI & TECH
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Patsnap Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Patsnap Eureka Blog
Learn More
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic, Popular Technical Reports.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap|About US| Contact US: help@patsnap.com