A data storage method and apparatus

Abstract

The embodiment of the present application provides a data storage method and device, when a remote read-write request of reading and writing storage data in a specified format according to a specified network protocol is received, a pre-implemented local read-write interface is called to read or write the storage data in the specified format in a pre-obtained shared memory; when a local read-write request of reading and writing storage data in a specified format is received, a pre-implemented local read-write interface is called to read or write the storage data in the specified format in a pre-obtained shared memory.

Description

Technical Field

[0001] This invention relates to the field of Internet technology, and in particular to a data storage method and apparatus. Background Technology

[0002] Currently, widely used memory-based key-value stores in the internet field include Memcache and Redis. Memcache is a free, open-source, high-performance, distributed memory object caching system; Redis is an open-source, memory-based key-value store database written in ANSI C, containing various data structures, supporting networking, and offering optional persistence. These stores typically store data in their own allocated memory and provide services externally via network protocols. Compared to some disk-based databases, these memory-based databases offer significantly higher performance. Memcache and Redis primarily provide services over the network, listening on a network port. Other applications access the database via the network. This leads to a problem: even if other applications and the database are on the same machine, they still need to access the database through the local network protocol stack, resulting in significant overhead and low efficiency. Furthermore, Memcache lacks persistence; if the database restarts or the machine experiences a problem and restarts, existing data will be lost, affecting the normal operation of other applications.

[0003] In the process of developing this invention, the applicant discovered at least the following problems in the prior art:

[0004] When a network protocol-based database supports access from both local and remote applications, accessing the database from a local application consumes significant resources and is relatively inefficient. Summary of the Invention

[0005] This invention provides a data storage method and apparatus that solves the problem that when a network protocol-based database supports access from both local and remote applications, local applications experience high resource consumption and low efficiency when accessing the network protocol-based database.

[0006] To achieve the above objectives, in one aspect, embodiments of the present invention provide a data storage method, comprising:

[0007] In response to a received remote read / write request for stored data in a specified format according to a specified network protocol, a pre-implemented local read / write interface is invoked to read or write the stored data in shared memory in the specified format.

[0008] In response to a received local read / write request for reading or writing stored data in the specified format, a pre-implemented local read / write interface is invoked to read or write the stored data in shared memory in the specified format;

[0009] The shared memory is memory allocated in the local system for one or more processes to read and write data in a shared manner.

[0010] Furthermore, it also includes:

[0011] Upon receiving the remote read / write request or the local read / write request, before reading or writing the stored data in the shared memory, it is determined whether the shared memory has been allocated.

[0012] If it is determined that the shared memory has not been allocated, then the shared memory is allocated in the local system according to the pre-specified size; and the shared memory is formatted according to the specified formatting rules.

[0013] If it is determined that the shared memory has already been allocated, then the shared memory that has passed the verification is obtained.

[0014] Furthermore, the shared memory includes header memory and at least one storage area;

[0015] Allocate the shared memory in the local system according to the pre-specified size; and format the shared memory according to the specified formatting rules, including:

[0016] Allocate the shared memory in the local system according to the size of the shared memory;

[0017] The number of storage areas that can be divided is calculated based on the start and end addresses of the space occupied by all the pre-specified storage areas, the number of sectors of each storage area, and the sector size determination factor.

[0018] Based on the header information that needs to be stored in the header memory of the shared memory and the number of storage areas, the size of the header memory that needs to be retained is calculated; the shared memory is divided into the header memory and the at least one storage area; wherein, the header information includes: the version of the shared memory, the size of the header memory, the size of the shared memory, the read-write lock, the starting address of the sector linked list corresponding to each storage area, and a bucket array of a specified array length;

[0019] Each storage area is formatted into at least one sector according to the specific sector byte number of that storage area; wherein, the specific sector byte number corresponding to each storage area is obtained according to the sector size determinant factor;

[0020] For each storage area, the addresses of all sectors in that storage area are stored in a linked list corresponding to that storage area, and this linked list is used as the sector linked list corresponding to that storage area.

[0021] The version of the shared memory, the size of the header memory, the size of the shared memory, the read-write lock, the starting address of the sector linked list corresponding to each storage area, and the empty bucket array are stored in the header memory;

[0022] The step of obtaining the verified shared memory includes:

[0023] Open the shared memory that has been allocated in the local system;

[0024] Read the version of the shared memory from the header memory;

[0025] The version of the shared memory is compared with the specified memory version. If the version of the shared memory is compatible with the specified memory version, the information that the shared memory is available is returned and the shared memory is obtained.

[0026] Each element of the bucket array is used to store the starting address of a linked list consisting of sectors that have been written with data in a specified format.

[0027] Furthermore, the specified network protocols include the Memcache protocol and the Redis protocol;

[0028] The specified format includes the key string and the value;

[0029] The invocation of the pre-implemented local read / write interface to read or set the stored data in shared memory according to the specified format includes:

[0030] The key string in the stored data is converted into a hash value using a hash algorithm; the hash value is divided by the length of the bucket array, and the remainder is obtained.

[0031] If a read operation is called using the local read / write interface, a read lock is acquired on the shared memory. The starting address of the linked list stored by the array element at the remainder position in the bucket array is found, and the found linked list is used as the linked list to be traversed. The elements of the linked list to be traversed are traversed. If the key string recorded in the sector corresponding to an element is found to be the same as the key string corresponding to the stored data, the value recorded in the sector corresponding to that element is returned. If no key string recorded in the sector corresponding to any element is found to be the same as the key string corresponding to the stored data, an error message indicating that the key string does not exist is returned.

[0032] If a write operation is called using the local read / write interface, a write lock is acquired on the shared memory, and the total space occupied by the key string and value of the stored data is calculated. The total space occupied is compared with the number of bytes in a specific sector of each storage area to determine the minimum number of bytes in a specific sector required to store the stored data. A free sector is searched in the sector list corresponding to the storage area with the minimum number of bytes in a specific sector. If no free sector is found, an error message indicating no available memory is returned. If a free sector is found, the key string and value of the stored data are copied to the free sector, the free sector is added to the linked list stored by the array element at the remainder position in the bucket array, and the free sector is marked as used.

[0033] Furthermore, it also includes:

[0034] According to the specified persistence rules, the local read / write interface is invoked to persist the data in the specified format in the shared memory to the persistent storage device; and upon restart, the data in the specified format is loaded from the persistent storage device into the shared memory.

[0035] On the other hand, embodiments of the present invention provide a data storage device, including:

[0036] The network read / write unit is used to respond to a received remote read / write request for reading or writing stored data in a specified format according to a specified network protocol, and to trigger the local read / write module to complete the call to the pre-implemented local read / write interface to read or write the stored data in the shared memory in the specified format.

[0037] The local read / write unit is used to respond to a received local read / write request for reading or writing stored data in the specified format, and trigger the local read / write module to complete the call to the pre-implemented local read / write interface to read or write the stored data in the shared memory in the specified format;

[0038] The shared memory is memory allocated in the local system for one or more processes to read and write data in a shared manner.

[0039] Furthermore, it also includes:

[0040] The memory detection unit is used to determine whether the shared memory has been allocated before reading or writing the stored data in the shared memory after receiving the remote read / write request or the local read / write request.

[0041] A memory creation unit is used to allocate the shared memory in the local system according to the pre-specified size of the shared memory if it is determined that the shared memory has not been allocated; and to format the shared memory according to a specified formatting rule.

[0042] The memory acquisition unit is used to acquire the verified shared memory if it is determined that the shared memory has been allocated.

[0043] Furthermore, the shared memory includes header memory and at least one storage area;

[0044] The memory creation unit includes:

[0045] A shared memory allocation module is used to allocate the shared memory in the local system according to the size of the shared memory.

[0046] The storage area quantity planning module is used to calculate the number of storage areas that can be divided based on the start and end addresses of the space occupied by all storage areas in advance, the number of sectors of each storage area and the sector size determination factor.

[0047] The first formatting module is used to calculate the size of the header memory to be retained based on the header information that needs to be stored in the header memory of the shared memory and the number of storage areas; and to divide the shared memory into the header memory and the at least one storage area; wherein, the header information includes: the version of the shared memory, the size of the header memory, the size of the shared memory, the read-write lock, the starting address of the sector linked list corresponding to each storage area, and a bucket array of a specified array length;

[0048] The second formatting module is used to format each storage area into at least one sector according to the specific sector byte number of the storage area; wherein, the specific sector byte number corresponding to each storage area is obtained according to the sector size determinant factor;

[0049] The third formatting module is used to store the addresses of all sectors in each storage area in a linked list corresponding to that storage area, and use that linked list as the sector linked list corresponding to that storage area.

[0050] The fourth formatting module is used to store the version of the shared memory, the size of the header memory, the size of the shared memory, the read-write lock, the starting address of the sector linked list corresponding to each storage area, and the empty bucket array in the header memory;

[0051] The memory acquisition unit is specifically configured to: open the shared memory already allocated in the local system; read the version of the shared memory from the header memory; compare the version of the shared memory with a specified memory version; if the version of the shared memory is compatible with the specified memory version, return information that the shared memory is available and acquire the shared memory;

[0052] Each element of the bucket array is used to store the starting address of a linked list consisting of sectors that have been written with data in a specified format.

[0053] Furthermore, the specified network protocols include the Memcache protocol and the Redis protocol;

[0054] The specified format includes the key string and the value;

[0055] The local read / write module includes:

[0056] The read / write position acquisition module is used to convert the key string in the stored data into a hash value using a hash algorithm; and to divide the hash value by the length of the bucket array and obtain the remainder.

[0057] The reading module is used to, if a read operation is called using the local read / write interface, acquire a read lock on the shared memory, find the starting address of the linked list stored in the array element at the remainder position in the bucket array, and use the found linked list as the linked list to be traversed; traverse the elements of the linked list to be traversed, and if it is found that the key string recorded in the sector corresponding to a certain element is the same as the key string corresponding to the stored data, return the value recorded in the sector corresponding to that element; if it is not found that the key string recorded in the sector corresponding to any element is the same as the key string corresponding to the stored data, return an error message that the key string does not exist.

[0058] The write module is used to acquire a write lock on the shared memory if a write operation is called using the local read / write interface, and calculate the total space length occupied by the key string and value of the stored data; compare the total space length with the number of bytes in a specific sector of each storage area to determine the minimum number of bytes in a specific sector required to store the stored data; search for a free sector in the sector linked list corresponding to the storage area with the minimum number of bytes in a specific sector; if no free sector is found, return an error message that no memory is available; if a free sector is found, copy the key string and value of the stored data to the free sector, add the free sector to the linked list stored by the array element at the remainder position in the bucket array, and mark the free sector as used.

[0059] Furthermore, it also includes:

[0060] The persistence unit is used to persist data in a specified format from the shared memory to a persistent storage device by calling the local read / write interface according to specified persistence rules; and to load the specified format data from the persistent storage device to the shared memory upon restart.

[0061] The above technical solution offers the following advantages: By using shared memory and providing a local read / write interface for shared memory, local applications can bypass the network protocol stack and directly interact with the memory area corresponding to the shared memory-based database, thereby achieving extremely high performance while remaining compatible with remote read / write requests using specified network protocols. Through the shared memory mechanism, data storage implemented using the same shared memory can be shared by multiple local applications and components simultaneously, and also supports remote access by multiple external users via specified network protocols. Furthermore, by persisting data according to specified persistence rules, data loss is guaranteed even during machine restarts. Attached Figure Description

[0062] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0063] Figure 1 This is a flowchart of a data storage method according to one embodiment of the present invention;

[0064] Figure 2 This is a flowchart illustrating one embodiment of the present invention of accessing shared memory via remote read / write requests;

[0065] Figure 3 This is a flowchart illustrating one embodiment of the present invention of accessing shared memory directly through a local read / write interface.

[0066] Figure 4 This is a schematic diagram of the data flow between the local read / write unit, the network read / write unit, the persistence unit, and the shared memory in one embodiment of the present invention.

[0067] Figure 5 This is a flowchart of a persistence process according to one embodiment of the present invention;

[0068] Figure 6 This is a schematic diagram of the structure of a data storage device according to one embodiment of the present invention. Detailed Implementation

[0069] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0070] On the one hand, such as Figure 1 As shown, an embodiment of the present invention provides a data storage method, including:

[0071] Step S100, as follows Figure 2 As shown, in response to a received remote read / write request for reading or writing stored data in a specified format according to a specified network protocol, a pre-implemented local read / write interface is invoked to read or write the stored data in shared memory in the specified format.

[0072] Step S101, as follows Figure 3 As shown, in response to a received local read / write request for reading or writing stored data in the specified format, a pre-implemented local read / write interface is invoked to read or write the stored data in shared memory in the specified format;

[0073] The shared memory is memory allocated in the local system for one or more processes to read and write data in a shared manner.

[0074] In some embodiments, by invoking local system calls or custom local shared memory allocation and acquisition interfaces, new shared memory is allocated in local memory or a handle to an already allocated shared memory is obtained, thus preparing for subsequent steps to access the shared memory. There are various methods for implementing shared memory, which can be selected according to specific needs and the specific system. For example, in Linux systems, this includes, but is not limited to, shared memory mapping via mmap, XSI shared memory, or POSIX shared memory. Pre-implemented local read / write interfaces are used to perform read and write operations on shared memory through local system calls or custom local shared memory operation interfaces, enabling the reading and writing of stored data in a specified format. The specified format of the data stored in shared memory can be determined according to specific requirements. For example, the specified format includes, but is not limited to, KeyValue (i.e., key string and value) pairs or binary format data customized according to requirements. When a remote read / write request is received, it is converted into a call to the local read / write interface. Remote read / write requests are used to communicate with data in local shared memory via a network interface. These requests can originate from an application on another computing device on the internet. Applications on the same computing device as the shared memory can choose to operate on the data in local shared memory via remote read / write requests using a specified network protocol, or preferably directly operate on the data in local shared memory through the local read / write interface. In specific implementation and deployment, such as... Figure 4As shown, the local read / write interface is pre-encapsulated as a local interface library module. This module supports local applications (i.e., local read / write units) in directly manipulating shared memory, and also supports the network interface module (i.e., network read / write unit) in converting remote read / write requests into calls to the local read / write interface before reading and writing to shared memory. The process of handling remote read / write requests for data stored in a specified format according to a specified network protocol is pre-encapsulated as a network interface module. Remote applications or local applications that wish to access data in shared memory via a specified network protocol can access the data in shared memory through the network interface module; local applications that wish to efficiently manipulate data in shared memory can directly access the data in shared memory through the local interface library module. Through the shared memory mechanism, the same shared memory data service can be provided to one or more remote applications and / or one or more local applications simultaneously. Specified network protocols include, but are not limited to, Memcache and Redis protocols.

[0075] The embodiments of this invention have the following technical advantages: By using shared memory and providing a local read / write interface for reading and writing shared memory, local applications can bypass the network protocol stack and directly interact with the memory area corresponding to the shared memory-based database, thereby achieving extremely high performance while being compatible with remote read / write requests using specified network protocols. Through the shared memory mechanism, data storage implemented based on the same shared memory can be shared by multiple local applications and components simultaneously, and also supports remote access by multiple external users via specified network protocols. Through compatibility with Memcache and Redis protocols, this embodiment can seamlessly replace the Memcache or Redis components already used in the current project, achieving the effect of improving the data read / write efficiency of local applications with minimal modification cost.

[0076] Furthermore, it also includes:

[0077] Upon receiving the remote read / write request or the local read / write request, before reading or writing the stored data in the shared memory, it is determined whether the shared memory has been allocated.

[0078] If it is determined that the shared memory has not been allocated, then the shared memory is allocated in the local system according to the pre-specified size; and the shared memory is formatted according to the specified formatting rules.

[0079] If it is determined that the shared memory has already been allocated, then the shared memory that has passed the verification is obtained.

[0080] In some embodiments, the system call of the specific operating system can be used to determine whether the required shared memory has been allocated. If not, a local system call or a custom shared memory allocation interface is called to allocate new shared memory according to the required size, and the shared memory is formatted according to the specified formatting rules. For example, the shared memory can be divided into a metadata area and a storage data area. The metadata area records how data is stored in the storage data area, and the storage data area stores the actual data to be stored. Another example is to use the entire shared memory as a storage area and write the actual data sequentially, etc. The formatting methods listed here are not intended to limit the formatting rules of the technical solution of this invention. Those skilled in the art can define specific formatting rules according to their own project needs. If the required shared memory has been allocated, the shared memory is opened, and its descriptor, handle, or pointer is obtained. The data in the shared memory is read, and necessary verifications are performed, such as whether the shared memory version matches the expected version, whether the data in the shared memory is corrupted or lost, etc. After successful verification, the descriptor, handle, or pointer of the shared memory is returned for local read / write interface operations on the shared memory. In specific implementations, such as... Figure 3 As shown, the steps described above in the embodiments of the present invention can be pre-deployed in various local applications; or as... Figure 2 As shown, the steps described above in this embodiment are pre-deployed in the network interface module; alternatively, the steps can be pre-deployed in both local applications and the network interface module. The first local application or network interface module to start will detect that shared memory is not allocated, and thus allocate the shared memory. Subsequent local applications or network interface modules that detect that the required shared memory has been allocated will directly open the shared memory. For example, when implementing the shared memory in this embodiment using the XSI shared memory method in a Linux system, a globally unique identifier can be defined to identify the shared memory in this embodiment. The shared memory is then operated in each local application and network interface module using this globally unique identifier and system calls related to XSI shared memory.

[0081] The embodiments of the present invention have the following technical effects: by determining whether shared memory exists, a new shared memory is created or an existing shared memory is opened, so that each local application and network interface module can open the same shared memory at the same time, achieving the effect that each local application and remote application can operate on data of a specified format through the same shared memory.

[0082] Furthermore, the shared memory includes header memory and at least one storage area;

[0083] The step of allocating shared memory in the local system according to the pre-specified size of the shared memory, and formatting the shared memory according to a specified formatting rule, includes:

[0084] Allocate the shared memory in the local system according to the size of the shared memory;

[0085] The number of storage areas that can be divided is calculated based on the start and end addresses of the space occupied by all the pre-specified storage areas, the number of sectors of each storage area, and the sector size determination factor.

[0086] Based on the header information that needs to be stored in the header memory of the shared memory and the number of storage areas, the size of the header memory that needs to be retained is calculated; the shared memory is divided into the header memory and the at least one storage area; wherein, the header information includes: the version of the shared memory, the size of the header memory, the size of the shared memory, the read-write lock, the starting address of the sector linked list corresponding to each storage area, and a bucket array of a specified array length;

[0087] Each storage area is formatted into at least one sector according to the specific sector byte number of that storage area; wherein, the specific sector byte number corresponding to each storage area is obtained according to the sector size determinant factor;

[0088] For each storage area, the addresses of all sectors in that storage area are stored in a linked list corresponding to that storage area, and this linked list is used as the sector linked list corresponding to that storage area.

[0089] The version of the shared memory, the size of the header memory, the size of the shared memory, the read-write lock, the starting address of the sector linked list corresponding to each storage area, and the empty bucket array are stored in the header memory;

[0090] The step of obtaining the verified shared memory includes:

[0091] Open the shared memory that has been allocated in the local system;

[0092] Read the version of the shared memory from the header memory;

[0093] The version of the shared memory is compared with the specified memory version. If the version of the shared memory is compatible with the specified memory version, the information that the shared memory is available is returned and the shared memory is obtained.

[0094] Each element of the bucket array is used to store the starting address of a linked list consisting of sectors that have been written with data in a specified format.

[0095] In some embodiments, depending on the specific system, the system's local system call or a custom local shared memory allocation interface is invoked to allocate shared memory according to the required size. When determining the size of the header memory, sufficient space can be reserved for the header memory by pre-specifying the starting address and ending address of the space occupied by all storage areas. The size of the available storage area space is obtained by subtracting the starting address from the ending address of the space occupied by all storage areas. The size of the available storage area space is determined by the sector size determination factor and the pre-specified minimum... The sector size in bytes determines the sector size in bytes of other storage areas. Specifically, for the Nth storage area, the sector size determinant factor raised to the power of (N-1) is multiplied by the minimum sector size in bytes to obtain the specific sector size in bytes of the Nth storage area. Multiplying the specific sector size in bytes of each storage area by the predefined number of sectors for each storage area yields the expected storage space required. By adjusting the number of storage areas to ensure that the expected storage space does not exceed the available storage space, the number of storage areas is obtained. Here, N is an integer greater than or equal to 1. The number of storage areas is obtained by finding the largest N value that ensures the expected storage space does not exceed the available storage space. For example, if the minimum sector is predefined as 32 bytes and the sector size determination factor is 2, then the sector size in the first storage area is 32 bytes, the sector size in the second storage area is 32 * 2 = 64 bytes, the sector size in the third storage area is 64 * 2 = 128 bytes, and so on. Multiplying the number of bytes in a specific sector of each storage area by the predefined number of sectors for each storage area yields the estimated storage space required. By adjusting the number of storage areas to ensure that the estimated storage space does not exceed the available storage space, the number of storage areas is obtained. The header information includes: the shared memory version, the size of the header memory, the size of the shared memory, the read-write lock, the starting address of the sector linked list corresponding to each storage area, and a bucket array of specified array length. Summing up the space occupied by the above parts gives the size of the header memory. Each storage area is deployed sequentially after the header memory, either adjacent to the header memory or with a reserved free area. For each storage area, it is divided into one or more sectors based on its specific sector byte count, and the address of each sector is obtained. These sectors are then organized into a linked list corresponding to that storage area. The shared memory version, the size of the header memory, the size of the shared memory, the read-write lock, the starting address of the sector linked list corresponding to each storage area, and an empty bucket array are stored in the header memory. The shared memory is then formatted. At this point, since the bucket array is empty, no data has been written to the shared memory.

[0096] When an existing shared memory is accessed, for example, if a memory detection unit, memory creation unit, and memory acquisition unit are deployed in the network interface module, the network interface module is started first. If no shared memory is available, the network interface module will create and format a new shared memory through the memory creation unit. Then, the local application starts. The local application also deploys a memory detection unit, a memory creation unit, and a memory acquisition unit. The local application finds that the shared memory already exists and needs to access it, verifying whether the shared memory version is compatible with the desired specified memory version. If compatible, the local application is allowed to continue using the shared memory, and can obtain parameters such as the shared memory descriptor, handle, or pointer for the local read / write interface to operate on the shared memory. If they are the same, and not compatible, the local application needs to terminate its access to the shared memory. In other embodiments, the local application may be started first, followed by the network interface module. The shared memory allocation and acquisition process under different startup orders can be understood based on the above embodiments. When determining version compatibility, it can be determined by comparing the shared memory version with the specified memory version. For example, if the shared memory version is equal to the specified memory version, it is compatible; if the shared memory version is greater than and / or equal to the specified memory version, it is compatible; if the shared memory version is less than and / or equal to the specified memory version, it is compatible; or if the specified memory version represents a version range, as long as the shared memory version is within this range, it is compatible.

[0097] The embodiments of the present invention have the following technical effects: the shared memory is automatically formatted according to the size of the shared memory and other pre-given parameters, and the shared memory is formatted into multiple storage areas according to different sector sizes. The appropriate storage area can be selected according to the specific data size to avoid memory waste, while still maintaining high memory read and write efficiency.

[0098] Furthermore, the specified network protocols include the Memcache protocol and the Redis protocol;

[0099] The specified format includes the key string and the value;

[0100] The invocation of the pre-implemented local read / write interface to read or set the stored data in shared memory according to the specified format includes:

[0101] The key string in the stored data is converted into a hash value using a hash algorithm; the hash value is divided by the length of the bucket array, and the remainder is obtained.

[0102] If a read operation is called using the local read / write interface, a read lock is acquired on the shared memory. The starting address of the linked list stored by the array element at the remainder position in the bucket array is found, and the found linked list is used as the linked list to be traversed. The elements of the linked list to be traversed are traversed. If the key string recorded in the sector corresponding to an element is found to be the same as the key string corresponding to the stored data, the value recorded in the sector corresponding to that element is returned. If no key string recorded in the sector corresponding to any element is found to be the same as the key string corresponding to the stored data, an error message indicating that the key string does not exist is returned.

[0103] If a write operation is called using the local read / write interface, a write lock is acquired on the shared memory, and the total space occupied by the key string and value of the stored data is calculated. The total space occupied is compared with the number of bytes in a specific sector of each storage area to determine the minimum number of bytes in a specific sector required to store the stored data. A free sector is searched in the sector list corresponding to the storage area with the minimum number of bytes in a specific sector. If no free sector is found, an error message indicating no available memory is returned. If a free sector is found, the key string and value of the stored data are copied to the free sector, the free sector is added to the linked list stored by the array element at the remainder position in the bucket array, and the free sector is marked as used.

[0104] In some embodiments, the specified network protocols include the Memcache protocol and the Redis protocol. For projects that have already used Memcache and Redis, the system implemented by this method can seamlessly replace the existing project without affecting other parts of the existing project, thereby achieving the goal of quickly updating each existing project and improving development efficiency. The key string in the stored data is converted into a hash value, such as a 32-bit integer, using a hash algorithm. This hash value is then modulo the length of the bucket array to obtain the remainder. Each element of the bucket array stores the starting address of a linked list consisting of sectors containing data of a specified format. The element at the remainder position in the bucket array stores the starting address of the linked list containing the stored data or about to be added. When reading the value of a specified key string from shared memory, a read-write lock is set on the header memory. This lock does not prevent other read operations but prevents other write operations. Based on the remainder, the starting address of the linked list containing the sector storing the key string and its value can be obtained. Each sector in this linked list is traversed. When the key string is found, the corresponding value is retrieved from that sector. If the key string is not found after traversal, it means that the key string and its value do not exist in the current shared memory, and an error message indicating that the key string does not exist is returned. When a write operation is called using the local read / write interface, a read / write lock is set in the header memory to lock the shared memory, prohibiting other read / write operations. The total space occupied by the key-value string and its value is calculated, and this total space is compared with the number of bytes in a specific sector of each storage area. During this comparison, the number of bytes occupied by control fields such as used flags or storage format auxiliary fields needs to be taken into account to determine the minimum number of bytes in a specific sector required to store the data. For example, if the total space occupied by the key-value string and its value is 29 bytes, and after formatting the shared memory, there are three storage areas, each with its own... The sector sizes are 16 bytes, 32 bytes, and 64 bytes respectively. Each sector requires 1 byte as a used identifier. At this point, it can be found that the sector size of the second storage area is the smallest sector size greater than 29 + 1 = 30 bytes. Therefore, it is determined to store the 29 bytes of data in a sector of the second storage area. A free sector is searched in the sector linked list corresponding to the second storage area. If a free sector is found, the 29 bytes of data are written to the found free sector, and the free sector is appended to the linked list at the aforementioned remainder position in the bucket array. The free sector is then marked as used.

[0105] The embodiments of this invention have the following technical effects: by distributing stored data across linked lists at different index positions of a bucket array using a hash algorithm, and by setting an appropriate specified array length value for the bucket array, the read and write efficiency of stored data can be improved, thus enhancing the read and write efficiency of shared memory. For example, using a larger specified array length for the bucket array can significantly reduce the length of each linked list to be traversed, reducing traversal time and improving read and write efficiency.

[0106] Furthermore, it also includes:

[0107] According to the specified persistence rules, the local read / write interface is invoked to persist the data in the specified format in the shared memory to the persistent storage device; and upon restart, the data in the specified format is loaded from the persistent storage device into the shared memory.

[0108] In some embodiments, such as Figure 5 As shown, when specified persistence rules are met, such as including but not limited to waiting for the amount of data stored in shared memory to reach a specified threshold or waiting for a specified timeout period since the last persistence operation, the local read / write interface is invoked to read the data stored in shared memory and write it to a persistent storage device, such as a disk, to ensure data retention even after power failure. After a system restart, a handle to the shared memory is obtained, and the persistent data file in the local persistent storage device is read. If persistent data exists, it is reloaded from the persistent storage device into the shared memory to continue providing remote and / or local requests.

[0109] On the other hand, such as Figure 6 As shown, an embodiment of the present invention provides a data storage device, including:

[0110] The network read / write unit 600 is used to respond to a received remote read / write request for reading and writing stored data in a specified format according to a specified network protocol, and trigger the local read / write module to complete the call of a pre-implemented local read / write interface to read or write the stored data in the shared memory in the specified format.

[0111] The local read / write unit 601 is used to respond to a received local read / write request for reading or writing stored data in the specified format, and trigger the local read / write module to complete the call of a pre-implemented local read / write interface to read or write the stored data in the shared memory in the specified format.

[0112] The shared memory is memory allocated in the local system for one or more processes to read and write data in a shared manner.

[0113] Furthermore, it also includes:

[0114] The memory detection unit is used to determine whether the shared memory has been allocated before reading or writing the stored data in the shared memory after receiving the remote read / write request or the local read / write request.

[0115] A memory creation unit is used to allocate the shared memory in the local system according to the pre-specified size of the shared memory if it is determined that the shared memory has not been allocated; and to format the shared memory according to a specified formatting rule.

[0116] The memory acquisition unit is used to acquire the verified shared memory if it is determined that the shared memory has been allocated.

[0117] Furthermore, the shared memory includes header memory and at least one storage area;

[0118] The memory creation unit includes:

[0119] A shared memory allocation module is used to allocate the shared memory in the local system according to the size of the shared memory.

[0120] The storage area quantity planning module is used to calculate the number of storage areas that can be divided based on the start and end addresses of the space occupied by all storage areas in advance, the number of sectors of each storage area and the sector size determination factor.

[0121] The first formatting module is used to calculate the size of the header memory to be retained based on the header information that needs to be stored in the header memory of the shared memory and the number of storage areas; and to divide the shared memory into the header memory and the at least one storage area; wherein, the header information includes: the version of the shared memory, the size of the header memory, the size of the shared memory, the read-write lock, the starting address of the sector linked list corresponding to each storage area, and a bucket array of a specified array length;

[0122] The second formatting module is used to format each storage area into at least one sector according to the specific sector byte number of the storage area; wherein, the specific sector byte number corresponding to each storage area is obtained according to the sector size determinant factor;

[0123] The third formatting module is used to store the addresses of all sectors in each storage area in a linked list corresponding to that storage area, and use that linked list as the sector linked list corresponding to that storage area.

[0124] The fourth formatting module is used to store the version of the shared memory, the size of the header memory, the size of the shared memory, the read-write lock, the starting address of the sector linked list corresponding to each storage area, and the empty bucket array in the header memory;

[0125] The memory acquisition unit is specifically configured to: open the shared memory already allocated in the local system; read the version of the shared memory from the header memory; compare the version of the shared memory with a specified memory version; if the version of the shared memory is compatible with the specified memory version, return information that the shared memory is available and acquire the shared memory;

[0126] Each element of the bucket array is used to store the starting address of a linked list consisting of sectors that have been written with data in a specified format.

[0127] Furthermore, the specified network protocols include the Memcache protocol and the Redis protocol;

[0128] The specified format includes the key string and the value;

[0129] The local read / write module includes:

[0130] The read / write position acquisition module is used to convert the key string in the stored data into a hash value using a hash algorithm; and to divide the hash value by the length of the bucket array and obtain the remainder.

[0131] The reading module is used to, if a read operation is called using the local read / write interface, acquire a read lock on the shared memory, find the starting address of the linked list stored in the array element at the remainder position in the bucket array, and use the found linked list as the linked list to be traversed; traverse the elements of the linked list to be traversed, and if it is found that the key string recorded in the sector corresponding to a certain element is the same as the key string corresponding to the stored data, return the value recorded in the sector corresponding to that element; if it is not found that the key string recorded in the sector corresponding to any element is the same as the key string corresponding to the stored data, return an error message that the key string does not exist.

[0132] The write module is used to acquire a write lock on the shared memory if a write operation is called using the local read / write interface, and calculate the total space length occupied by the key string and value of the stored data; compare the total space length with the number of bytes in a specific sector of each storage area to determine the minimum number of bytes in a specific sector required to store the stored data; search for a free sector in the sector list corresponding to the storage area with the minimum number of bytes in a specific sector; if no free sector is found, return an error message that no memory is available; if a free sector is found, copy the key string and value of the stored data to the free sector, add the free sector to the list stored by the array element at the remainder position in the bucket array, and remove the free sector from the sector list corresponding to the storage area with the minimum number of bytes in a specific sector.

[0133] Furthermore, it also includes:

[0134] The persistence unit is used to persist data in a specified format from the shared memory to a persistent storage device by calling the local read / write interface according to specified persistence rules; and to load the specified format data from the persistent storage device to the shared memory upon restart.

[0135] The data storage device provided in this embodiment of the invention is a product-type embodiment that corresponds one-to-one with the aforementioned data storage method. Those skilled in the art can understand the embodiment of the data storage device based on the aforementioned data storage method embodiment, and will not be described again here.

[0136] The embodiments of this invention have the following technical advantages: By using shared memory and providing a local read / write interface for reading and writing shared memory, local applications can bypass the network protocol stack and directly interact with the memory area corresponding to the shared memory-based database, thereby achieving extremely high performance while being compatible with remote read / write requests using specified network protocols. Through the shared memory mechanism, data storage implemented based on the same shared memory can be shared by multiple local applications and components simultaneously, and also supports remote access by multiple external users via specified network protocols. Furthermore, by persisting data according to specified persistence rules, data loss is guaranteed even during machine restarts.

[0137] The technical solutions of the present invention will be described in detail below with reference to specific application examples. For technical details not described in the implementation process, please refer to the relevant descriptions above.

[0138] The embodiments of the present invention include four modules: shared memory provided by the server's own memory, network interface module, local library interface module, and data persistence module.

[0139] The relationships between the various modules can be found in the following reference. Figure 4 The arrows in the diagram indicate the data flow between modules. Since the modules are loosely coupled, there is no explicit data flow between them except for sharing a common memory segment. Each module can work independently.

[0140] The shared memory is provided by the server's own shared memory function; the network interface module provides network-based services to the outside world through listening TCP ports, and the service protocols include Memcache and Redis protocols; the persistence module is used to persist the contents of the shared memory to the disk according to specified rules, and load the data from the disk into the shared memory upon restart.

[0141] For the local interface library module, its working method and flowchart can be simply described as follows: Figure 3 .

[0142] For the network interface module, its working method and flowchart can be simply described as follows: Figure 2 :

[0143] For the "open shared memory handle or request shared memory" phase of the above two modules, the main process is as follows:

[0144] 1. Check if the shared memory block exists. If it does not exist, go to step 2; otherwise, go to step 6.

[0145] 2. Invoke a system call to create a shared memory block;

[0146] 3. Calculate the header memory size META_SIZE that needs to be reserved based on the shared memory size;

[0147] 3.1 Calculate the required number of SLABs, slabcount, based on MIN_SLAB_SIZE, MAX_SLAB_SIZE, and SLAB_FACTOR;

[0148] 3.2 Let HEADER_SIZE be the storage space required for the head of a linked list, VERSION_SIZE be the storage space for the shared memory version number, and LOCK_SIZE be the storage space for the read-write lock. Then the formula for calculating META_SIZE is:

[0149] META_SIZE=VERSION_SIZE+LOCK_SIZE+slabcount*HEADER_SIZE+len(buckets)*HEADER_SIZE;

[0150] 4. Starting from META_SIZE onwards, the shared memory is divided into several small SLABs (sectors) according to the configuration, from MIN_SLAB_SIZE (the starting address of the space occupied by all pre-specified storage areas) to MAX_SLAB_SIZE (the ending address of the space occupied by all pre-specified storage areas). The space occupied by all pre-specified storage areas can be divided into several small SLABs (sectors) at a pace of SLAB_FACTOR (sector size determinant, preferably 2). The memory addresses of these small blocks are grouped by SLAB_SIZE (the number of bytes in each specific sector of each storage area) and stored in a linked list (that is, for each storage area, the addresses of all sectors in that storage area are stored in a linked list corresponding to that storage area, and this linked list is used as the sector linked list for that storage area).

[0151] 5. Store the current library version (i.e., the shared memory version), META_SIZE (i.e., the size of the header memory), the total size of the shared memory, the addresses of all SLAB linked lists in step 4 (i.e., the starting addresses of the respective sector linked lists corresponding to each storage area), and an empty buckets array (i.e., the bucket array) into the memory area between 0 and META_SIZE (i.e., the header memory), and go to step 10.

[0152] 6. Invoke a system call to open shared memory;

[0153] 7. Read the first 16 bytes of shared memory (corresponding to the shared memory version and the size of the header memory) to obtain the library version and META_SIZE information;

[0154] 8. Compare library versions (i.e., compare the shared memory version with the specified memory version). If they are inconsistent, exit; otherwise, continue.

[0155] 9. Read the memory from the beginning of memory to META_SIZE (i.e., read the information in the header memory) and restore all metadata information, where the metadata information is used to access the header information stored in the header memory in the form of structured variables;

[0156] 10. Done.

[0157] For the memory operation phase of the above two modules, the main process is as follows:

[0158] 1. Obtain the key (i.e., the key string) to be operated on, and use a hash algorithm to convert the key into a 32-bit hash value.

[0159] 2. Divide the hash value by the length of the buckets array in META (i.e., the header memory) (i.e., the specified array length), and get the remainder N;

[0160] 3. Obtain the operation type (i.e., determine the type of the local read / write interface being called). If it is a get type (i.e., a read operation), proceed to step 4; otherwise, proceed to step 5.

[0161] 4. Acquire a read lock and find the linked list at the Nth position in the buckets array (i.e., the linked list to be traversed). Traverse this linked list. If the value of the key recorded in the linked list is the same as the key value in the read request, return the corresponding value (i.e., the value corresponding to the key string). If the key string is not found after traversal, return the NO_KEY error (i.e., the error message that the key string does not exist).

[0162] 5. If it is a write operation, then get the value (i.e. the value corresponding to the key), and calculate the total length of the key and value, TOTAL_LEN (i.e. the total space occupied by the key string and the value used to store the data);

[0163] 6. Based on TOTAL_LEN, calculate the most suitable SLAB_SIZE in ascending order; (that is, compare the total occupied space length with the number of bytes in a specific sector of each storage area to determine the minimum number of bytes in a specific sector required to store the stored data);

[0164] 7. From the linked list corresponding to the most suitable SLAB_SIZE, search for an empty item (i.e., search for a free sector in the linked list of sectors corresponding to the storage area corresponding to the minimum specific sector size). If not found, return a NO_MEM error (i.e., no available memory). If found, copy the key and value to the memory area of ​​this item (i.e., if a free sector is found, copy the key string and value of the stored data to that free sector). Add this memory area to the linked list at the Nth position in the buckets array. Mark the sector as used. Return OK.

[0165] For the persistence module, its working method and flowchart can be simply described as follows: Figure 6 .

[0166] Of the four modules mentioned above, apart from shared memory, which is implemented and required by the operating system, the other modules can be combined arbitrarily and play a role in specific scenarios. For example, when all modules are enabled, it can provide KV services that are much faster than cross-machine storage for local key-value storage needs, and at the same time provide TCP ports for services on other machines to use. Applications on two machines can share the same data through different usage methods. In addition, the persistence module will periodically flush the data in memory to the disk to ensure that data will not be lost even if the machine loses power.

[0167] A typical use case is as follows: an application A depends on some key-value data (such as the mapping between UID and username). This application only reads this key-value data, while the original data is pushed by another application B. In this scenario, the network interface module is first enabled, memory is allocated, and a TCP port is listened to. At this time, application B can start pushing key-value data via the TCP protocol. At the same time, application A starts and directly reads the key-value data pushed by application B through the local library interface module without network protocol loss. Similarly, in order to solve the problem that application B needs to re-push data when the machine restarts, the persistence module will periodically flush the memory to disk to ensure that the data is not lost after the machine restarts.

[0168] The technical solution of this invention has the following technical effects: In the prior art, for key-value databases such as Redis and Memcache, the logic of application interaction with them is roughly as follows: 1. The application creates a TCP connection with the key-value database; 2. The application sends an interaction request through the TCP connection; 3. The server receives the request from the TCP connection; 4. The server processes the request (converting the request into read / write operations on memory); 5. The server returns the result to the client through the TCP connection; 6. The application obtains the result from the TCP connection.

[0169] Regarding the technical solution of this invention, local applications can directly read and write memory through the local library interface module, thereby bypassing the TCP interaction process, that is, bypassing the network protocol stack. Furthermore, it supports modular assembly, allowing multiple programs and components to share the same memory area simultaneously. It also supports TCP protocol implementation to allow external users to access the memory, and supports persistence operations according to certain rules, ensuring that data is not lost even when the machine restarts.

[0170] Compared to existing key-value databases such as Redis and Memcache, the technical solution of this invention allows programs to bypass the network protocol stack and directly interact with the memory area corresponding to the key-value database, thereby achieving extremely high performance.

[0171] It should be understood that the specific order or hierarchy of steps in the disclosed process is an example of an exemplary method. Based on design preferences, it should be understood that the specific order or hierarchy of steps in the process may be rearranged without departing from the scope of this disclosure. The appended method claims provide elements of various steps in an exemplary order and are not intended to limit the scope to the specific order or hierarchy described.

[0172] In the above detailed description, various features are combined together in a single embodiment to simplify this disclosure. This approach to disclosure should not be construed as reflecting an intention that embodiments of the claimed subject matter require more features than are explicitly stated in each claim. Rather, as reflected in the appended claims, the invention is presented with fewer features than all of the features of the single disclosed embodiment. Therefore, the appended claims are hereby explicitly incorporated into the detailed description, wherein each claim stands alone as a preferred embodiment of the invention.

[0173] The disclosed embodiments have been described above to enable any person skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be applied to other embodiments without departing from the spirit and scope of this disclosure. Therefore, this disclosure is not limited to the embodiments given herein, but is consistent with the broadest scope of the principles and novel features disclosed in this application.

[0174] The foregoing description includes examples of one or more embodiments. It is certainly impossible to describe all possible combinations of components or methods in order to describe the above embodiments, but those skilled in the art will recognize that further combinations and arrangements of the various embodiments are possible. Therefore, the embodiments described herein are intended to cover all such changes, modifications, and variations that fall within the scope of the appended claims. Furthermore, the term "comprising" as used in the specification or claims is interpreted in a manner similar to the term "including," just as "including:" is interpreted as a conjunction in the claims. Additionally, the use of any term "or" in the specification of the claims is intended to mean "non-exclusive or."

[0175] Those skilled in the art will also understand that the various illustrative logical blocks, units, and steps listed in the embodiments of the present invention can be implemented by electronic hardware, computer software, or a combination of both. To clearly demonstrate the interchangeability of hardware and software, the functions of the various illustrative components, units, and steps described above have been generally described. Whether such functionality is implemented through hardware or software depends on the specific application and the overall system design requirements. Those skilled in the art can implement the described functions using various methods for each specific application, but such implementation should not be construed as exceeding the scope of protection of the embodiments of the present invention.

[0176] The various illustrative logic blocks or units described in the embodiments of this invention can be implemented or operate the described functions using a general-purpose processor, digital signal processor, application-specific integrated circuit (ASIC), field-programmable gate array or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof. The general-purpose processor can be a microprocessor; alternatively, it can be any conventional processor, controller, microcontroller, or state machine. The processor can also be implemented using a combination of computing devices, such as a digital signal processor and a microprocessor, multiple microprocessors, one or more microprocessors combined with a digital signal processor core, or any other similar configuration.

[0177] The steps of the methods or algorithms described in the embodiments of this invention can be directly embedded in hardware, a software module executed by a processor, or a combination of both. The software module can be stored in RAM, flash memory, ROM, EPROM, EEPROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium in the art. Exemplarily, the storage medium can be connected to the processor so that the processor can read information from and write information to the storage medium. Optionally, the storage medium can also be integrated into the processor. The processor and storage medium can be housed in an ASIC, which can be housed in a user terminal. Optionally, the processor and storage medium can also be housed in different components of the user terminal.

[0178] In one or more exemplary designs, the functions described in the embodiments of the present invention can be implemented in hardware, software, firmware, or any combination of these three. If implemented in software, these functions can be stored on a computer-readable medium or transmitted on a computer-readable medium in the form of one or more instructions or code. Computer-readable media include computer storage media and communication media that facilitate the transfer of computer programs from one place to another. Storage media can be any available media that can be accessed by a general-purpose or special-purpose computer. For example, such computer-readable media can include, but is not limited to, RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to carry or store program code in the form of instructions or data structures and other forms that can be read by a general-purpose or special-purpose computer, or a general-purpose or special-purpose processor. Furthermore, any connection can be suitably defined as a computer-readable medium, for example, if the software is transmitted from a website, server or other remote resource via a coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL) or wirelessly, such as infrared, wireless and microwave, it is also included in the defined computer-readable medium. The disks and discs mentioned include compressed disks, laser discs, optical discs, DVDs, floppy disks, and Blu-ray discs. Disks typically copy data magnetically, while disks typically copy data optically using lasers. Combinations of the above can also be contained in computer-readable media.

[0179] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A data storage method, characterized in that, include: In response to a received remote read / write request for stored data in a specified format according to a specified network protocol, a pre-implemented local read / write interface is invoked to read or write the stored data in shared memory in the specified format. In response to a received local read / write request for reading or writing stored data in the specified format, a pre-implemented local read / write interface is invoked to read or write the stored data in shared memory in the specified format; The shared memory is memory allocated in the local system for one or more processes to read and write data in a shared manner; The method further includes: after receiving the remote read / write request or the local read / write request, before reading or writing the stored data in the shared memory, determining whether the shared memory has been allocated; If it is determined that the shared memory has not been allocated, then the shared memory is allocated in the local system according to the pre-specified size; and the shared memory is formatted according to the specified formatting rules. If it is determined that the shared memory has already been allocated, then the shared memory that has passed the verification is obtained; The shared memory is allocated by the first local application or network interface module to be started, and the subsequently started local application or network interface module directly opens the shared memory if it is detected that the shared memory has been allocated.

2. The data storage method as described in claim 1, characterized in that, The shared memory includes header memory and at least one storage area; Allocate the shared memory in the local system according to the pre-specified size of the shared memory; And format the shared memory according to the specified formatting rules, including: Allocate the shared memory in the local system according to the size of the shared memory; The number of storage areas that can be divided is calculated based on the start and end addresses of the space occupied by all the pre-specified storage areas, the number of sectors of each storage area, and the sector size determination factor. Based on the header information that needs to be stored in the header memory of the shared memory and the number of storage areas, the size of the header memory that needs to be retained is calculated; the shared memory is divided into the header memory and the at least one storage area; wherein, the header information includes: the version of the shared memory, the size of the header memory, the size of the shared memory, the read-write lock, the starting address of the sector linked list corresponding to each storage area, and a bucket array of a specified array length; Each storage area is formatted into at least one sector according to the specific sector byte number of that storage area; wherein, the specific sector byte number corresponding to each storage area is obtained according to the sector size determinant factor; For each storage area, the addresses of all sectors in that storage area are stored in a linked list corresponding to that storage area, and this linked list is used as the sector linked list corresponding to that storage area. The version of the shared memory, the size of the header memory, the size of the shared memory, the read-write lock, the starting address of the sector linked list corresponding to each storage area, and the empty bucket array are stored in the header memory; The step of obtaining the verified shared memory includes: Open the shared memory that has been allocated in the local system; Read the version of the shared memory from the header memory; The version of the shared memory is compared with the specified memory version. If the version of the shared memory is compatible with the specified memory version, the information that the shared memory is available is returned and the shared memory is obtained. Each element of the bucket array is used to store the starting address of a linked list consisting of sectors that have been written with data in a specified format.

3. The data storage method as described in claim 2, characterized in that, The specified network protocols include the Memcache protocol and the Redis protocol; The specified format includes the key string and the value; The invocation of the pre-implemented local read / write interface to read or set the stored data in shared memory according to the specified format includes: The key string in the stored data is converted into a hash value using a hash algorithm; the hash value is divided by the length of the bucket array, and the remainder is obtained. If a read operation is called using the local read / write interface, a read lock is acquired on the shared memory. The starting address of the linked list stored by the array element at the remainder position in the bucket array is found, and the found linked list is used as the linked list to be traversed. The elements of the linked list to be traversed are traversed. If the key string recorded in the sector corresponding to an element is found to be the same as the key string corresponding to the stored data, the value recorded in the sector corresponding to that element is returned. If no key string recorded in the sector corresponding to any element is found to be the same as the key string corresponding to the stored data, an error message indicating that the key string does not exist is returned. If a write operation is called using the local read / write interface, a write lock is acquired on the shared memory, and the total space occupied by the key string and value of the stored data is calculated. The total space occupied is compared with the number of bytes in a specific sector of each storage area to determine the minimum number of bytes in a specific sector required to store the stored data. A free sector is searched in the sector list corresponding to the storage area with the minimum number of bytes in a specific sector. If no free sector is found, an error message indicating no available memory is returned. If a free sector is found, the key string and value of the stored data are copied to the free sector, the free sector is added to the linked list stored by the array element at the remainder position in the bucket array, and the free sector is marked as used.

4. The data storage method as described in claim 1, characterized in that, Also includes: According to the specified persistence rules, the local read / write interface is invoked to persist the data in the specified format in the shared memory to the persistent storage device; Upon restart, the specified format data is loaded from the persistent storage device into the shared memory.

5. A data storage device, characterized in that, include: The network read / write unit is used to respond to a received remote read / write request for reading or writing stored data in a specified format according to a specified network protocol, and to trigger the local read / write module to complete the call to the pre-implemented local read / write interface to read or write the stored data in the shared memory in the specified format. The local read / write unit is used to respond to a received local read / write request for reading or writing stored data in the specified format, and trigger the local read / write module to complete the call to the pre-implemented local read / write interface to read or write the stored data in the shared memory in the specified format; The shared memory is memory allocated in the local system for one or more processes to read and write data in a shared manner; The memory detection unit is used to determine whether the shared memory has been allocated before reading or writing the stored data in the shared memory after receiving the remote read / write request or the local read / write request. A memory creation unit is used to allocate the shared memory in the local system according to the pre-specified size of the shared memory if it is determined that the shared memory has not been allocated; and to format the shared memory according to a specified formatting rule. A memory acquisition unit is used to acquire the verified shared memory if it is determined that the shared memory has been allocated. The shared memory is allocated by the first local application or network interface module to be started, and the subsequently started local application or network interface module directly opens the shared memory if it is detected that the shared memory has been allocated.

6. The data storage device as described in claim 5, characterized in that, The shared memory includes header memory and at least one storage area; The memory creation unit includes: A shared memory allocation module is used to allocate the shared memory in the local system according to the size of the shared memory. The storage area quantity planning module is used to calculate the number of storage areas that can be divided based on the start and end addresses of the space occupied by all storage areas in advance, the number of sectors of each storage area and the sector size determination factor. The first formatting module is used to calculate the size of the header memory to be retained based on the header information that needs to be stored in the header memory of the shared memory and the number of storage areas; and to divide the shared memory into the header memory and the at least one storage area; wherein, the header information includes: the version of the shared memory, the size of the header memory, the size of the shared memory, the read-write lock, the starting address of the sector linked list corresponding to each storage area, and a bucket array of a specified array length; The second formatting module is used to format each storage area into at least one sector according to the specific sector byte number of the storage area; wherein, the specific sector byte number corresponding to each storage area is obtained according to the sector size determinant factor; The third formatting module is used to store the addresses of all sectors in each storage area in a linked list corresponding to that storage area, and use that linked list as the sector linked list corresponding to that storage area. The fourth formatting module is used to store the version of the shared memory, the size of the header memory, the size of the shared memory, the read-write lock, the starting address of the sector linked list corresponding to each storage area, and the empty bucket array in the header memory; The memory acquisition unit is specifically configured to: open the shared memory already allocated in the local system; read the version of the shared memory from the header memory; compare the version of the shared memory with a specified memory version; if the version of the shared memory is compatible with the specified memory version, return information that the shared memory is available and acquire the shared memory; Each element of the bucket array is used to store the starting address of a linked list consisting of sectors that have been written with data in a specified format.

7. The data storage device as described in claim 6, characterized in that, The specified network protocols include the Memcache protocol and the Redis protocol; The specified format includes the key string and the value; The local read / write module includes: The read / write position acquisition module is used to convert the key string in the stored data into a hash value using a hash algorithm; and to divide the hash value by the length of the bucket array and obtain the remainder. The reading module is used to, if a read operation is called using the local read / write interface, acquire a read lock on the shared memory, find the starting address of the linked list stored in the array element at the remainder position in the bucket array, and use the found linked list as the linked list to be traversed; traverse the elements of the linked list to be traversed, and if it is found that the key string recorded in the sector corresponding to a certain element is the same as the key string corresponding to the stored data, return the value recorded in the sector corresponding to that element; if it is not found that the key string recorded in the sector corresponding to any element is the same as the key string corresponding to the stored data, return an error message that the key string does not exist. The write module is used to acquire a write lock on the shared memory if a write operation is called using the local read / write interface, and calculate the total space length occupied by the key string and value of the stored data; compare the total space length with the number of bytes in a specific sector of each storage area to determine the minimum number of bytes in a specific sector required to store the stored data; search for a free sector in the sector linked list corresponding to the storage area with the minimum number of bytes in a specific sector; if no free sector is found, return an error message that no memory is available; if a free sector is found, copy the key string and value of the stored data to the free sector, add the free sector to the linked list stored by the array element at the remainder position in the bucket array, and mark the free sector as used.

8. The data storage device as described in claim 5, characterized in that, Also includes: The persistence unit is used to persist data in a specified format in the shared memory to a persistent storage device by calling the local read / write interface according to the specified persistence rules. Upon restart, the specified format data is loaded from the persistent storage device into the shared memory.

Images