A skip list-based sharded shared memory storage method

Abstract

The application discloses a kind of based on skip list's fragmented shared memory storage method, the storage system includes fragmented management group, fragmented module, shared memory module;The fragmented management group is used to open or create management group shared memory module, load all existing fragmented module;The fragmented module realizes the principle of skip list, and the data stored in shared memory module is stored and accessed;The shared memory module is used to manage fragmented module module information, in fragmented management component, the default maximum value of fragmented module, the number of stored data, fragmented read-write lock, fragmented list are stored.This application creates shared memory, uses the retrieval mode of skip list, automatically increases the fragmented data storage according to the size of storage data amount, expands the maximum threshold of storage data amount;Since shared memory is used, the access and retrieval time reaches nanosecond level;Support high-concurrency data fast access retrieval.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of shared memory, in particular to a shared memory storage method based on skip list. BACKGROUND

[0002] Shared memory refers to a large-capacity memory that can be accessed by different central processing units (CPUs) in a multi-processor computer system. Since multiple CPUs need to access the memory quickly, the memory needs to be cached. After any cached data is updated, the shared memory needs to be updated immediately, otherwise different processors may use different data.

[0003] Redis (Remote Dictionary Server) is an in-memory cache database written in C language, and its data model is key-value. Redis can support a variety of data types such as String, List, Hash, Set, and Sorted Set, and is widely used.

[0004] For data storage, a data storage system such as redis is generally used. Redis uses a key-value storage method, uses a key value as a key, and provides a unique index for data storage query, so that the corresponding stored data can be queried. However, the query speed is slow, and when the data storage capacity increases and exceeds the maximum threshold of the redis system, the maximum threshold of the stored data needs to be increased. Therefore, the present application is developed. SUMMARY

[0005] In view of the problems in the prior art, the present application provides a shared memory storage method based on skip list.

[0006] In order to achieve the above purpose, the present application provides a shared memory storage system based on skip list, which comprises a shard management group, a shard module and a shared memory module.

[0007] The shard management group is used to open or create a management group shared memory module, and load all existing shard modules.

[0008] The shard module realizes the principle of skip list to store and access data stored in the shared memory module.

[0009] The shared memory module is used for managing the sharding module information, and the default maximum value of the sharding module, the storage data quantity, the sharding read-write lock and the sharding list are stored in the sharding management component.

[0010] Preferably, the skip list in the sharding module is in the form of a multi-level single-item linked list, and the key values are stored in the skip list in ascending order, and the key values in the skip list are stored in order.

[0011] The application also provides a sharding shared memory storage method based on a skip list, comprising the following steps:

[0012] Step S1: inserting data:

[0013] Step S101: inserting data key-value, locking the read-write lock of the sharding management group, and checking whether the data quantity of all sharding modules has reached or exceeded 95% of the total capacity;

[0014] Step S102: if yes, automatically adding a sharding module, searching the sharding module and the position for data insertion, and releasing the read-write lock;

[0015] Step S103: if no, directly searching the sharding module and the position for data insertion, and releasing the read-write lock;

[0016] Step S2: data query:

[0017] Step S201: locking the read-write lock of the sharding management group, searching the key values of the skip lists of each sharding module, comparing the key values to be searched with the key values of the first data of the skip list headers of the sharding modules, and querying the sharding module where the key values are located;

[0018] Step S202: comparing the data key values in the skip list with the key values to be searched, and querying the data if the key values are equal, and releasing the read-write lock;

[0019] Step S3: data deletion:

[0020] Step S301: locking the read-write lock of the sharding management group, searching the key values of the skip list headers of each sharding module, comparing the key values to be deleted with the key values of the skip list headers, and querying the sharding module where the key values are located;

[0021] Step S302: comparing the data key values in the skip list with the data to be deleted; if the key values are equal, deleting the corresponding nodes in the skip list structure and releasing the corresponding shared memory, and releasing the read-write lock; otherwise, directly releasing the read-write lock.

[0022] Preferably, the size of the newly added sharding module in step S102 is the default sharding size.

[0023] Preferably, the specific process of retrieving the inserted fragment and position of data in steps S102 and S103 is: retrieving the value of the key value of the skip list of each fragment, comparing the inserted key value with the key value of the first data of the skip list header, if it is smaller, the data is inserted into the previous fragment module, otherwise the data is inserted into the last fragment module, and the read-write lock is released.

[0024] Preferably, the specific method of querying the fragment where the key value is located in step S201 is: if the key value to be retrieved is smaller than the key value of the first data of the skip list header, the data is in the previous fragment, and the data is retrieved in the skip list chain table of the previous fragment, otherwise, the data is retrieved in the skip list chain table of the last fragment.

[0025] Preferably, in step S202, if the key value of the data in the skip list is not equal to the queried key value, it means that the data does not exist.

[0026] Preferably, the specific method of querying the fragment where the key value is located in step S301 is: if the key value of the data to be deleted is smaller than the key value of the skip list header, the data is in the previous fragment, and the data is retrieved in the skip list chain table of the previous fragment; otherwise, the data is retrieved in the last fragment.

[0027] The technical scheme of the present application has the following beneficial effects:

[0028] The present application creates shared memory and uses skip list retrieval method, automatically increases the data storage fragments according to the size of the stored data, and expands the maximum threshold of the stored data. Since shared memory is used, the access and retrieval time reaches nanosecond level. High-concurrency data fast access and retrieval are supported.

[0029] The present application is based on skip list data storage, and the data is stored in shared memory, the data query efficiency reaches nanosecond level, the data query speed is fast, the local multi-process high-concurrency access is supported, the data storage is flexibly managed by fragmentation, for the data storage with small data volume in the early stage, the machine can select a server with lower memory, the price of the server can be lower, when the data volume increases, the memory size of the server can be directly upgraded, without any operation, the fragmented data can be flexibly increased, and the maximum storage capacity of the data can be increased.

[0030] The present application is used for local persistent storage of information data, and flexibly expands the storage size of the storage system. BRIEF DESCRIPTION OF DRAWINGS

[0031] Figure 1 The association diagram of the fragment management group, the fragment module and the shared memory module of the present application;

[0032] Figure 2 is a skip list structure diagram of the present application;

[0033] Figure 3 is a data insertion flow of the present application;

[0034] Figure 4 is a data query flow of the present application;

[0035] Figure 5 is a data deletion flow of the present application. DETAILED DESCRIPTION

[0036] The present application is further illustrated below in conjunction with the accompanying drawings and specific embodiments.

[0037] Referring to Figures 1 to 5 The present application provides a skip list-based sharded shared memory storage system, comprising a shard management group, a shard module and a shared memory module.

[0038] The shard management group is used to open or create a management group shared memory module and load all existing shard modules; the shard module implements the principle of a skip list to store and access data stored in the shared memory module; and the shared memory module is used to manage shard module information, and stores the default maximum value of the shard module, the number of stored data, the shard read-write lock and the shard list in the shard management component. The shard module is provided with multiple shards, and the shared memory module is provided with multiple shared memories, each shard corresponding to one shared memory.

[0039] The shard management group implements management of all shared memory shards, including shard creation, opening, data retrieval, deletion and insertion functions. The shard management group creates a specific name shared memory, which is used to manage shard information. In the shard management group, the default maximum value of the shard, the number of stored data, the shard read-write lock and the shard list are stored. The shard management group first initializes the shared memory for managing shard information. First, the specific name shared memory is opened. If the shared memory does not exist, the shared memory is created. The shared memory for managing information stores the name of the shard management group and the default size of each shard.

[0040] The shard implements the principle of a skip list to store and access data stored in the shared memory. The skip list in the shard is in the form of a multi-level single-item linked list. In order to speed up the retrieval efficiency, the key values are stored in the skip list in ascending order, and the key values in the skip list are stored in order.

[0041] The present application also provides a skip list-based sharded shared memory storage method, comprising the following steps:

[0042] Step S1: inserting data:

[0043] Step S101: insert data key-value, lock the read-write lock of the shard management group, and check whether the data volume of all shards has reached or exceeded 95% of the total capacity; the size of the newly added shard module in the step S101 is the default shard size.

[0044] Step S102: if yes, automatically add a shard module, and retrieve the shard and position of the data insertion, and release the read-write lock.

[0045] Step S103: if no, directly retrieve the shard and position of the data insertion, and release the read-write lock.

[0046] The specific process of retrieving the shard and position of the data insertion in the step S102 and the step S103 is as follows: retrieve the key value of the skip list of each shard, compare the inserted key value with the key value of the first data of the skip list header, if it is smaller, the data is inserted into the previous shard module, otherwise, the data is inserted into the last shard module, and the read-write lock is released.

[0047] Step S2: data query:

[0048] Step S201: lock the read-write lock of the shard management group, retrieve the key value of the skip list of each shard, compare the key value to be retrieved with the key value of the first data of the skip list header, and query the shard where the key value is located; the specific method of querying the shard where the key value is located in the step S201 is as follows: if the key value to be retrieved is smaller than the key value of the first data of the skip list header, the data is in the previous shard, select the previous shard to retrieve the data in the skip list chain table, otherwise, select the last shard to retrieve the data in the skip list chain table.

[0049] Step S202: compare the data key value in the skip list with the key value to be retrieved, and if they are equal, the data is queried, and the read-write lock is released; if they are not equal, it means that the data does not exist.

[0050] Step S3: data deletion:

[0051] Step S301: lock the read-write lock of the shard management group, retrieve the key value of the skip list header of each shard, compare the data key value to be deleted with the key value of the skip list header, and query the shard where the key value is located; if the data key value to be deleted is smaller than the key value of the skip list header, the data is in the previous shard, select the previous shard to retrieve the data in the skip list chain table; otherwise, select the last shard to retrieve the data.

[0052] Step S302: compare the data key value in the skip list with the data to be deleted; if they are equal, delete the corresponding node in the skip list structure and release the corresponding shared memory, and release the read-write lock; otherwise, directly release the read-write lock.

[0053] The above merely provides the preferred embodiments of the present application, and is not intended to limit the patent scope of the present application. Any equivalent structure variations made according to the present application description and the attached drawings, or direct / indirect application in other related technical fields, shall be included in the patent protection scope of the present application.

Claims

1. A skip list based sharded shared memory storage method, characterized in that, It comprises the following steps: Step S1: inserting data: Step S101: inserting data key-value, locking the read-write lock of the shard management group, and checking whether the data volume of all shards has reached or exceeded 95% of the total capacity; Step S102: if yes, automatically adding a shard module, and retrieving the shard and position of data insertion, and releasing the read-write lock; Step S103: if no, directly retrieving the shard and position of data insertion, and releasing the read-write lock; Step S2: data query: Step S201: locking the read-write lock of the shard management group, retrieving the key value of the skip list of each shard, comparing the key value to be retrieved with the key value of the first data of the skip list head, and querying the shard where the key value is located; Step S202: comparing the data key value in the skip list with the key value to be retrieved, and querying the data if they are equal, and releasing the read-write lock; Step S3: data deletion: Step S301: locking the read-write lock of the shard management group, retrieving the key value of the skip list head of each shard, comparing the data key value to be deleted with the key value of the skip list head, and querying the shard where the key value is located; Step S302: comparing the data key value in the skip list with the data to be deleted; If they are equal, deleting the corresponding node in the skip list structure and releasing the corresponding shared memory, and releasing the read-write lock; otherwise, directly releasing the read-write lock; The specific process of retrieving the shard and position of data insertion in steps S102 and S103 is: retrieving the key value of the skip list of each shard, comparing the inserted key value with the key value of the first data of the skip list head, and if it is smaller, inserting the data into the previous shard module, otherwise inserting it into the last shard module, and releasing the read-write lock; The specific method of querying the shard where the key value is located in step S201 is: if the key value to be retrieved is smaller than the key value of the first data of the skip list head, the data is in the previous shard, and the previous shard is selected to retrieve the data in the skip list chain table; otherwise, the last shard is selected to retrieve the data in the skip list chain table; The specific method of querying the shard where the key value is located in step S301 is: if the data key value to be deleted is smaller than the key value of the skip list head, the data is in the previous shard, and the previous shard is selected to retrieve the data in the skip list chain table; otherwise, the last shard is selected to retrieve the data; The shard management group is used to open or create a shared memory module of the management group, and load all existing shard modules; The shard module realizes storage and access of data stored in the shared memory module based on the principle of skip list; In the shard management component, the default maximum value of the shard module, the number of stored data, the shard read-write lock, and the shard list are stored; The skip list in the shard module is in the form of a multi-level single-item chain table, and the key values are stored in the skip list in ascending order, and the key values in the skip list are stored in order. 2.The skip list based sharded shared memory storage method of claim 1, wherein, The size of the newly added shard module in step S102 is the default shard size.

3. The skip list based sharded shared memory storage method of claim 1, wherein, In step S202, if the data key value of the skip list is not equal to the queried key value, it means that the data does not exist.

Images