Data storage method and device, computer equipment, readable storage medium and program product

Abstract

The invention relates to a data storage method and device, computer equipment, a readable storage medium and a program product. The method comprises the following steps: determining index information of to-be-stored data through first address offset based on a memory page, storing the index information and the data to a data block in the memory page, sending the memory page to a storage thread, and storing the to-be-stored data by the storage thread according to the index information stored by the data block in the memory page. Compared with a traditional mode of transmitting a small amount of data in single-time transmission, according to the scheme, the data blocks are constructed in the memory page, the data needing to be stored and the index information of the data needing to be stored are stored in the memory page, the data are more compactly stored in the memory page of single-time transmission in the storage process, and therefore the data storage efficiency is improved.

Description

Technical Field

[0001] The present application relates to the field of data storage technology, and in particular to a data storage method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Art

[0002] Against the backdrop of rapidly expanding data transmission demands, efficient data transmission solutions have become a hot research topic. Currently, computers need to store process data during operation, such as in a KVDB (key-value database). In an all-flash array framework, this can be achieved by storing a certain amount of data at a time when storing process key-value pairs. However, the current storage of process data is limited within the transmission space available for a single storage session, making it difficult to store the data compactly, reducing data storage efficiency.

[0003] Therefore, the current storage method for key-value pair data has the defect of low storage efficiency. Summary of the Invention

[0004] Based on this, it is necessary to provide a data storage method, device, computer equipment, computer-readable storage medium and computer program product that can improve storage efficiency in response to the above technical problems.

[0005] In a first aspect, the present application provides a data storage method, comprising:

[0006] Get the data to be stored and the first address offset of the memory page;

[0007] determining index information of the data according to the first address offset of the memory page;

[0008] storing the index information and the data in a data block; wherein the data block is stored in the memory page;

[0009] The memory page is sent to a storage thread, and the storage thread is used to store the data according to the index information stored in the data block in the memory page.

[0010] In this embodiment, the server determines the index information of the data to be stored based on the first address offset of the memory page, stores the index information and data in a data block within the memory page, and sends the memory page to the storage thread. The storage thread then stores the data to be stored based on the index information stored in the data block within the memory page. Compared to the traditional method of transmitting a small amount of data in a single transmission, this solution constructs data blocks within the memory page to store the data to be stored and its index information. This allows the data to be stored more compactly within the memory page that is transmitted in a single transmission during the storage process, thereby improving data storage efficiency.

[0011] In one embodiment, the step of obtaining the first address offset of the memory page includes:

[0012] If the memory page is the first memory page among multiple memory pages, a first address offset of the memory page is determined according to a memory base address.

[0013] In this embodiment, the server connects the various memory pages by setting a message header in the memory page and setting the address offset of the next memory page in the message header. The location of each memory page can be obtained through the memory base address, thereby improving the efficiency of obtaining memory pages.

[0014] In one embodiment, the step of obtaining the first address offset of the memory page includes:

[0015] If the memory page is not the first memory page among the multiple memory pages, the first address offset of the memory page is determined according to a message header in a previous memory page corresponding to the memory page.

[0016] In this embodiment, the server connects the various memory pages by setting a message header in the memory page and setting the address offset of the next memory page in the message header. The location of each memory page can be obtained through the message header and the memory base address, thereby improving the efficiency of obtaining memory pages.

[0017] In one embodiment, determining the index information of the data according to the first address offset of the memory page includes:

[0018] Obtaining the data length of the data;

[0019] Determining a second address offset corresponding to the data according to the first address offset of the memory page and the data length;

[0020] The index information of the data is obtained according to the second address offset and the data length.

[0021] In this embodiment, the server establishes index information of the data based on the first address offset of the memory page, so that the data can be stored more compactly in the data block during the transmission process of data storage, thereby achieving the technical effect of improving data storage efficiency.

[0022] In one embodiment, the data block includes an index area and a data area;

[0023] Storing the index information and the data in a data block includes:

[0024] storing the second address offset and the data length in the index information of the data in the index area in the data block;

[0025] storing the data in the data area of ​​the data block;

[0026] The data block storing the index information and the data is obtained according to the index area including the second address offset and the data length, and the data area including the data.

[0027] In this embodiment, the server sets up a data block containing an index area and a data area, stores the index information of the data and the data itself in the index area and data area of ​​the data block respectively, and stores multiple data blocks in the memory page. The data can be placed in the memory page more compactly, and the performance is also higher. In the case of cross-control, the processing delay can be reduced to the millisecond level, thereby improving the efficiency of data storage.

[0028] In one embodiment, storing the data according to the index information stored in the data block in the memory page includes:

[0029] For each of the data blocks in the memory page, determining a second address offset and a data length corresponding to the index area in the data block;

[0030] According to the second address offset and the data length, the corresponding data is obtained from the data area of ​​the data block for storage.

[0031] In this embodiment, the server implements fast search and analysis of data blocks in memory pages through the index area and data area in the data block, combined with the second address offset and data length of the data, thereby improving data storage efficiency.

[0032] In one embodiment, an offset record is stored in the data block; the offset record represents a third address offset of a next data block of the data block;

[0033] After acquiring the corresponding data from the data area of ​​the data block and storing the data according to the second address offset and the data length, the method further includes:

[0034] Determining a third address offset of the next data block according to the offset record in the data block;

[0035] Obtaining the next data block according to the third address offset;

[0036] According to the index information stored in the next data block, the data in the next data block is stored, and the step of determining the third address offset of the next data block according to the offset record in the data block is returned to execute until the data in each data block in the memory page is stored.

[0037] In this embodiment, the server can connect the various data blocks in the memory page through offset records. When the server traverses the memory page and performs data storage, it can quickly find and parse the data blocks, thereby improving the efficiency of data storage.

[0038] In a second aspect, the present application further provides a data storage device, comprising:

[0039] An acquisition module, configured to acquire data to be stored and a first address offset of a memory page;

[0040] a determination module, configured to determine index information of the data according to the first address offset of the memory page;

[0041] A first storage module is configured to store the index information and the data in a data block; the data block is stored in the memory page;

[0042] The second storage module is used to send the memory page to a storage thread, and the storage thread is used to store the data according to the index information stored in the data block in the memory page.

[0043] In a third aspect, the present application further provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above method when executing the computer program.

[0044] In a fourth aspect, the present application also provides a computer-readable storage medium having a computer program stored thereon, which implements the steps of the above method when executed by a processor.

[0045] In a fifth aspect, the present application also provides a computer program product, comprising a computer program, which implements the steps of the above method when executed by a processor.

[0046] The data storage method, apparatus, computer device, computer-readable storage medium, and computer program product described above determine index information for the data to be stored based on the first address offset of a memory page, store the index information and data in a data block within the memory page, and send the memory page to a storage thread, which then stores the data to be stored based on the index information stored in the data block within the memory page. Compared to the traditional method of transmitting a smaller amount of data in a single transmission, this solution constructs data blocks within the memory page to store the data to be stored and its index information, allowing the data to be more compactly stored within the memory page transmitted in a single transmission during the storage process, thereby improving data storage efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the following briefly introduces the drawings required for use in the embodiments of the present application or related technical descriptions. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying any creative work.

[0048] Figure 1 A schematic diagram of a flow chart of a data storage method in one embodiment;

[0049] Figure 2 is a schematic diagram of remote direct memory access in one embodiment;

[0050] Figure 3 A schematic diagram of the structure of a memory page in one embodiment;

[0051] Figure 4 Schematic diagram of the structure of a data block in one embodiment;

[0052] Figure 5 A schematic flow chart of a data storage method according to another embodiment;

[0053] Figure 6 is a structural block diagram of a data storage device in one embodiment;

[0054] Figure 7 FIG. 1 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0055] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0056] In related technologies, KV (Key-Value) values ​​can be stored in a corresponding KVDB (Key Value Database). As a storage management framework, KVDB assumes the functions and responsibilities of data storage and sorting. However, not all modules requiring storage can have a built-in KVDB. To ensure data cleanliness and independence, KVDB is fixed to the LKV (Local Key-Value) process within the CS (Client-Server) architecture. The KVIF (Key-Value Interface) transfers data from modules requiring storage to the LKV. The LKV parses the data according to the protocol and writes it to the KVDB based on the operations performed. The LKV within the CS has only single-threaded resources. To more efficiently process KV data storage requests, KVIF provides batch operations, significantly improving KV storage performance while reducing the frequency and pressure of network calls.

[0057] However, current data transmission technologies focus more on data security than on data volume and performance. Due to issues with data transmission methods and data arrangement, data transmission methods often rely on networks, resulting in lengthy links and significant performance losses. Furthermore, data cannot be optimally formatted according to the data format. Consequently, data cannot be stored more compactly within a single transmission, resulting in insufficient data volume in a single transmission and requiring multiple transmissions. This reduces performance, and the data volume per transmission is also smaller due to the inability to store data compactly.

[0058] In view of this, this solution determines the index information of the data to be stored based on the first address offset of the memory page, stores the index information and data in a data block within the memory page, and sends the memory page to the storage thread. The storage thread then stores the data to be stored based on the index information stored in the data block within the memory page. Compared to the traditional method of transmitting a small amount of data in a single transmission, this solution constructs data blocks within the memory page to store the data to be stored and its index information. This allows the data to be stored more compactly within the memory page that is transmitted in a single transmission during the storage process, thereby improving data storage efficiency.

[0059] In one embodiment, Figure 1As shown, a data storage method is provided. This embodiment uses the method applied to a server as an example. It is understood that the method can also be applied to terminals, which can be, but are not limited to, various personal computers, laptops, smart phones, tablet computers, Internet of Things devices, and portable wearable devices. Internet of Things devices can be smart speakers, smart TVs, smart air conditioners, smart car devices, etc. It can also be applied to a system including a terminal and a server, and is implemented through the interaction between the terminal and the server, including the following steps S202 to S208. Among them:

[0060] Step S202: Obtain the data to be stored and the first address offset of the memory page.

[0061] The data to be stored may be data that needs to be stored in the memory of the server, for example, a key-value pair.

[0062] Because different frameworks or scenarios have different requirements for data formats and performance, data transmission must be customized to optimize data transmission in different scenarios. This solution can store data based on the All-Flash framework and KVDB data structures.

[0063] Memory pages are the fundamental unit of operating system memory management. They divide physical memory into fixed-size blocks and map them to virtual memory through page tables, supporting process isolation and efficient memory allocation. An address offset is a relative position from the base address of a memory page or data structure. It is used to locate the storage location of specific data or resources in memory.

[0064] Among them, such as Figure 2 As shown, Figure 2 The figure is a schematic diagram of remote direct memory access in an embodiment. In order to take into account the security, data volume and performance of data transmission, when writing to the memory address, the network end uses the RDMA (Remote Direct Memory Access) technology through the KVIFC (Key-Value Interface Controller) process to write the memory address. Among them, the KVIFC process is a process for managing key-value pair storage, which is usually responsible for implementing the access, indexing and caching functions of key-value data. On the CS side, also called the server, the memory address can be obtained through RDMA. RDMA technology uses memory mapping technology to ensure the performance and security of data transmission between processes through memory mapping. KVIF ensures the performance of data transmission through efficient data arrangement and combination. Through the organic combination of the two solutions, the security, data volume and performance of data transmission are guaranteed at the same time.

[0065] The server can use an address offset to search for a memory page. Specifically, the server can obtain data to be stored and a first address offset of the memory page. The server can include one or more memory pages, and the first address offset can refer to the address offset of the memory page. The first address offset can be determined based on a memory base address or the address offset of a previous memory page corresponding to the memory page.

[0066] Step S204: determining index information of the data according to the first address offset of the memory page.

[0067] The server may generate index information corresponding to the data to be stored. The data may be stored in the form of index information. The index information represents key metadata for quickly locating and accessing the data. The server may determine the index information for the data based on the first address offset of the memory page. That is, the server may determine the index information for the data based on the first address offset of the memory page.

[0068] Step S206: storing the index information and the data into a data block; the data block is stored in the memory page.

[0069] The aforementioned memory page may include a Block (data block), which represents a management structure for storing KVs. The Block contains an information header that stores the KV information and data location within the Block, allowing for easier KV access and usage. Multiple Blocks may exist on a single memory page, and the server points to the next Block based on the NextOffset (offset record) in the Block information header. NextOffset refers to the relative position offset of the next data block, which is used to quickly locate subsequent data in a continuous storage structure. The server can then store the aforementioned index information and data in the aforementioned data block. The data block uses index information to store data, which can improve the compactness of data storage and, in turn, increase data storage and transmission capacity.

[0070] Step S208: sending the memory page to a storage thread, and the storage thread is used to store the data according to the index information stored in the data block in the memory page.

[0071] The server is provided with a storage thread, and the server sends the above-mentioned memory page to the storage thread. After obtaining the memory page, the storage thread obtains the specific data content of the data to be stored based on the index information stored in each data block in the memory page, and then stores the obtained data. For example, the server traverses each data block in the above-mentioned memory page through the storage thread, thereby obtaining the data to be stored from the data block and storing the data in the data block. The above-mentioned memory page stores multiple data blocks, and the data to be stored in each data block is compactly stored through index information, thereby increasing the amount of data stored in a single transmission.

[0072] In the above-described data storage method, index information for the data to be stored is determined based on the first address offset of the memory page. The index information and data are stored in a data block within the memory page. The memory page is then sent to a storage thread, which then stores the data to be stored based on the index information stored in the data block within the memory page. Compared to traditional methods that transfer smaller amounts of data in a single transmission, this solution constructs data blocks within the memory page to store the data to be stored and its index information. This allows the data to be more compactly stored within the memory page that is transferred in a single transmission during the storage process, thereby improving data storage efficiency.

[0073] In one embodiment, the step of obtaining the first address offset of the memory page includes: if the memory page is the first memory page among multiple memory pages, determining the first address offset of the memory page according to a memory base address.

[0074] In this embodiment, the server may store a plurality of memory pages, and the memory page may be a communication memory page. The memory page may determine the address offset starting from the memory base address. The memory base address may be a fixed address. The server may determine the first address offset of the memory page in combination with the location of the memory page. For example, the server may detect whether the memory page is the first memory page of multiple memory pages in the server. If so, the server may obtain the memory base address and determine the first address offset of the memory page based on the memory base address. The server may determine the corresponding first address offset based on the memory base address and the size of the memory page. The memory pages in the server may be connected together. For a memory page that is not the first memory page among multiple memory pages, the server may determine the first address offset in other ways.

[0075] In one embodiment, the step of obtaining the first address offset of the memory page includes: if the above-mentioned memory page is not the first memory page among the multiple above-mentioned memory pages, then determining the first address offset of the above-mentioned memory page based on the message header in the previous memory page corresponding to the above-mentioned memory page.

[0076] In this embodiment, the non-first memory page refers to a memory page that is not the first memory page among multiple memory pages; the message header refers to a message header set in the memory page, and the message header records the offset of the starting address of the next memory page connected to the memory page, that is, the first address offset. When the memory page is not the first memory page among multiple memory pages in the server, in order to determine the first address offset of the non-first memory page, the server can determine the first address offset of the memory page based on the message header of the previous memory page corresponding to the memory page.

[0077] Specifically, if Figure 3 As shown, Figure 3 The following is a schematic diagram of the memory page structure in one embodiment. The unit of data transmission in the server is a memory page, which is an 8KB memory page. This memory page can be a communication memory page, which is a portion of memory within a memory segment with a fixed size and address, used for scenarios such as inter-process communication or inter-controller communication. When provided to memory users, the communication memory is divided into multiple memory pages at an 8KB granularity; users will use 8KB memory pages.

[0078] When transferring and storing data in memory, the server needs to fit the data into 8KB communication memory pages in a reasonable and compact manner. When communication memory is needed for data storage, the server can calculate the number of 8KB memory pages n required based on the total amount of data passed in by the user, and then request n 8KB memory pages. The server can then link these n 8KB memory pages together and set a page chaining message header in each memory page, using the message header to link the 8KB pages together.

[0079] Because 8KB communication memory pages are sent to different controllers, each with a different machine pointer, but the memory page's offset from the communication memory base address is fixed, the server can record the offset of the next 8KB memory page's starting address in the message header of each 8KB memory page, that is, the first address offset of the next memory page. These pages are managed and stored in an SGL (Single Group List). The server parses the SGL through the receiving controller or process to obtain the 8KB memory page.

[0080] Through the above embodiment, the server sets a message header in the memory page and sets the address offset of the next memory page in the message header, thereby connecting the various memory pages. The location of each memory page can be obtained through the message header and the memory base address, thereby improving the efficiency of obtaining memory pages.

[0081] Because communication memory is a critical resource, the aforementioned data needs to be compactly packed into 8KB communication memory pages to conserve memory and improve memory utilization. Therefore, compacting KV data into 8KB communication memory pages requires indexing different information. The server then uses LKV to retrieve KV data based on the index information and writes it to the KVDB.

[0082] In one embodiment, determining the index information of the above-mentioned data based on the first address offset of the above-mentioned memory page includes: obtaining the data length of the above-mentioned data; determining the second address offset corresponding to the above-mentioned data based on the first address offset of the above-mentioned memory page and the above-mentioned data length; and obtaining the index information of the above-mentioned data based on the above-mentioned second address offset and the above-mentioned data length.

[0083] In this embodiment, the data length represents the length of the data to be stored, which can be determined based on the size of the data. The data can be stored in a corresponding data block in a memory page, so that the server can determine the second address offset corresponding to the data based on the first address offset and the data length of the memory page. The server obtains index information for the data based on the second address offset and the data length.

[0084] For example, for each piece of data, the server indexes the data to the second address offset corresponding to the first address offset of the memory page based on the data length, and records the length corresponding to the data.

[0085] Through this embodiment, the server establishes index information of the data based on the first address offset of the memory page, so that the data can be stored more compactly in the data block during the transmission process of data storage, thereby achieving the technical effect of improving data storage efficiency.

[0086] In one embodiment, the index information and the data are stored in a data block, including: storing the second address offset and the data length in the index information of the data in the index area of ​​the data block; storing the data in the data area of ​​the data block; and obtaining the data block storing the index information and the data based on the index area including the second address offset and the data length, and the data area including the data.

[0087] In this embodiment, the above-mentioned index information can be stored in the corresponding data block in the memory page. The above-mentioned data block may be provided with an index area and a data area. The index area is used to store the index information, and the data area is used to store the specific data that needs to be stored. After the server determines the second address offset of the above-mentioned data and determines the data length, the server can store the above-mentioned second address offset and the above-mentioned data length in the above-mentioned index information of the data to the above-mentioned index area in the above-mentioned data block, and store the above-mentioned data to the above-mentioned data area of ​​the above-mentioned data block. Thus, the server obtains the data block storing the index information and data based on the above-mentioned index area and data area. The above-mentioned second address offset and data length are recorded in the index area of ​​the above-mentioned data block, and the above-mentioned data area records the above-mentioned data to be stored.

[0088] Specifically, the server encapsulates data through KVIF in the KVIFC process. KVIFC provides thread resources and fixes thread calls to ensure the security and stability of thread resources. Each KVIFC process must ensure unique KVIFC information. Before upgrading or performing operations, the server stores KVIFC information in LKV. The server must carry KVIFC information when performing modifications to ensure data security. The server ensures data and resource security by binding threads to KVIF and combining the KVIFC information mechanism. When storing data, the server uses compact KV data encapsulation technology to ensure sufficient data volume at the sender.

[0089] The structure of the data block is as follows Figure 4 As shown, Figure 4 The following is a schematic diagram of the structure of a data block in an embodiment. The data block includes an index area and a data area. The index area stores the second address offset and data length corresponding to the data, so that the server can find the location of each data through the index area. The data area stores each specific data, such as Figure 4 The KVITEM (key-value data) in the above data area can store multiple key-value data. The server indexes each data, finds the address offset of the data based on the 8KB communication memory page, records the corresponding length of the data, and places the index at the end of the message header of the 8KB page.

[0090] In order to better organize the above data, the server sets up a container in the memory page, that is, the above data block Block, which is divided into an index area and a data area. Figure 4As shown, a data block can store multiple data items, and a memory page can store one or more data blocks, allowing for more compact data storage. Since memory pages are transferred to different controllers, the pointers of different controllers point to different internal and external locations. To ensure data consistency between different controllers, the server can use address offsets as the location data for the above pages, data blocks, and data. For example, if the data is KV data, the server can use data blocks to store KV data more compactly and accurately. By recording the offset and length of the KV data in the data blocks, the data can be securely read from different controllers.

[0091] Through this embodiment, the server sets up a data block containing an index area and a data area, stores the index information of the data and the data itself in the index area and data area of ​​the data block respectively, and stores multiple data blocks in the memory page. The data can be placed in the memory page more compactly, and the performance is also higher. In the case of cross-control, the processing delay can be reduced to the millisecond level, thereby improving the efficiency of data storage.

[0092] In one embodiment, the data is stored according to the index information stored in the data block in the memory page, including: for each data block in the memory page, determining the second address offset and data length corresponding to the index area in the data block; and obtaining the corresponding data from the data area of ​​the data block according to the second address offset and the data length for storage.

[0093] In this embodiment, when storing data, the server can obtain the corresponding data from the data block for storage. The above-mentioned memory page may include multiple data blocks, and the data block includes an index area and a data area. For each data block in the memory page, the server can obtain the second address offset and data length corresponding to the data in the index area of ​​the above-mentioned data block, so that the server can obtain the corresponding data from the data area corresponding to the data block based on the second address offset and data length for storage. Multiple data blocks are stored in the above-mentioned memory page, and each data block can also be provided with an offset record. The server implements the storage of data in each data block in the memory page by traversing.

[0094] In one embodiment, after obtaining the corresponding data from the data area of ​​the data block for storage based on the second address offset and the data length, it also includes: determining the third address offset of the next data block based on the offset record in the data block; obtaining the next data block based on the third address offset; storing the data in the next data block based on the index information stored in the next data block, and returning to execute the step of determining the third address offset of the next data block based on the offset record in the data block, until the data in each data block in the memory page is stored.

[0095] In this embodiment, the location of a data block in the memory page can be represented by a third address offset, and the data blocks can be connected via an offset record. The offset record set in the data block can record the third address offset of the next data block to be connected. The third address offset represents the offset of the data block relative to the base address of the memory page.

[0096] When the server stores data, after completing the storage of the data in a data block, it needs to find the next data block. At this time, the server can determine the third address offset of the next data block based on the offset record in the above data block, and obtain the next data block based on the above third address offset. Each data block is provided with an index area and a data area. The server can then store the data in the next data block based on the index information stored in the next data block and continue to traverse the various data blocks in the memory page, for example, returning to execute the above step of determining the third address offset of the next data block based on the above offset record in the above data block, until the data in each data block in the above memory page is stored, thereby completing the traversal process of each data block in the memory page.

[0097] Specifically, the data may be key-value data, or KV data. The memory page may store multiple KV data items. The offset recorded in the data block is also called NextOffset, which refers to the relative position offset of the next data block. The server records the third address offset of the next data block in the NextOffset of the data block, connecting the data blocks through the NextOffset. The storage thread may also be called the LKV thread, and the server uses the LKV thread to traverse each data block in the memory page. For example, the server uses the LKV thread to parse and obtain the number of memory pages, then loops through the memory pages, parsing the data blocks within the memory pages, retrieving and storing the data therein. The server then determines the location of the next data block based on the NextOffset of the data block and continues parsing the data blocks and storing the KV data until the server detects that the offset of the next data block recorded in NextOffset is 0, i.e., the offset pointer points back to the address of the memory page. The server then determines that the current memory page traversal is complete and can then retrieve the next memory page for traversal, parsing, and storage.

[0098] It should be noted that the server stores KV data compactly, and does not limit the amount of data transmitted by KV data. The transmission method will not be a limiting factor for the amount of data transmitted. In some embodiments, KV data transmission can use all space in the communication memory as a carrier.

[0099] Through the above embodiment, the server can connect the various data blocks in the memory page through NextOffset. When the server traverses the memory page and performs data storage, it can achieve fast search and analysis of data blocks, thereby improving the efficiency of data storage.

[0100] In an exemplary embodiment, Figure 5 As shown, Figure 5 This is a flow chart of a data storage method according to another embodiment. This embodiment includes the following steps:

[0101] Step S302: When communication memory is needed for data storage, the number n of 8K memory pages required for the data is calculated based on the total amount of data input by the user, and n 8K memory pages are applied for. These n 8K memory pages are connected and combined together, and a page chain message header is set in each memory page, and the 8K pages are connected together with the message header.

[0102] Among them, since the 8K communication memory page will be transmitted to different controllers, different machine pointers are different, but the offset of the memory page based on the communication memory base address is fixed, the server can record the offset of the starting address of the next 8K memory page in the message header of each 8K memory page, that is, the first address offset of the next memory page.

[0103] Step S304: index each data to find the address offset of the data based on the 8k communication memory page, record the length corresponding to the data, and put the index at the end of the message header of the 8k page.

[0104] To better organize this data, the server sets up containers within memory pages, known as data blocks. These blocks are divided into an index area and a data area. A data block can store multiple pieces of data, and a memory page can store one or more data blocks, enabling more compact data storage. Because memory pages are transferred to different controllers, pointers to different internal and external locations in different controllers point to different locations. To ensure data consistency across different controllers, the server uses address offsets as the location data for the pages, data blocks, and data.

[0105] Step S306, obtain the number of memory pages through LKV thread analysis, and loop through the memory pages, parse the data blocks in the memory pages, obtain the data therein and store them; according to the NextOffset of the data block, obtain the position of the next data block, and continue the process of parsing the data blocks and storing KV data until the server detects that the offset of the next data block recorded in NextOffset is 0, that is, when the offset pointer points back to the address of the memory page, it is determined that the current memory page traversal is completed, and the next memory page is obtained for traversal and parsing storage.

[0106] The memory page can store multiple KV data. The offset record in the data block is also called NextOffset, which refers to the relative position offset of the next data block. The server records the third address offset of the next data block in the NextOffset of the data block, connecting the data blocks through NextOffset. The storage thread can also be called the LKV thread. The server uses the LKV thread to traverse each data block in the memory page.

[0107] Through the above embodiment, the server builds data blocks in memory pages to store the data to be stored and its index information, so that the data is stored more compactly in the memory pages that are transmitted once during the storage process, thereby improving the efficiency of data storage.

[0108] It should be understood that, although the various steps in the flowcharts involved in the various embodiments described above are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the flowcharts involved in the various embodiments described above can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of steps or stages in other steps.

[0109] Based on the same inventive concept, embodiments of the present application further provide a data storage device for implementing the aforementioned data storage method. The solution provided by this device is similar to the solution described in the aforementioned method. Therefore, the specific limitations of one or more data storage device embodiments provided below can be found in the above-described limitations of the data storage method and are not further elaborated here.

[0110] In an exemplary embodiment, Figure 6 As shown, a data storage device is provided, including: an acquisition module 500, a determination module 502, a first storage module 504 and a second storage module 506, wherein:

[0111] The acquisition module 500 is used to acquire data to be stored and a first address offset of a memory page.

[0112] The determination module 502 is configured to determine index information of the data according to the first address offset of the memory page.

[0113] The first storage module 504 is configured to store the index information and the data into a data block; the data block is stored in the memory page.

[0114] The second storage module 506 is configured to send the memory page to a storage thread, and the storage thread is configured to store the data according to the index information stored in the data block in the memory page.

[0115] In one embodiment, the acquisition module 500 is configured to determine a first address offset of the memory page according to a memory base address if the memory page is the first memory page among multiple memory pages.

[0116] In one embodiment, the acquisition module 500 is configured to determine the first address offset of the memory page based on a message header in a previous memory page corresponding to the memory page if the memory page is not the first memory page among the multiple memory pages.

[0117] In one embodiment, the determination module 502 is used to obtain the data length of the data; determine the second address offset corresponding to the data based on the first address offset of the memory page and the data length; and obtain index information of the data based on the second address offset and the data length.

[0118] In one embodiment, the first storage module 504 is used to store the second address offset and the data length in the index information of the data into the index area in the data block; store the data into the data area of ​​the data block; and obtain the data block storing the index information and the data based on the index area including the second address offset and the data length, and the data area including the data.

[0119] In one embodiment, the second storage module 506 is used to determine, for each of the data blocks in the memory page, the second address offset and data length corresponding to the index area in the data block; and obtain the corresponding data from the data area of ​​the data block for storage based on the second address offset and the data length.

[0120] In one embodiment, the above-mentioned device also includes: a traversal module, which is used to determine the third address offset of the next data block according to the above-mentioned offset record in the above-mentioned data block; obtain the next data block according to the above-mentioned third address offset; store the data in the next data block according to the index information stored in the next data block, and return to execute the above-mentioned step of determining the third address offset of the next data block according to the above-mentioned offset record in the above-mentioned data block, until the data in each data block in the above-mentioned memory page is stored.

[0121] Each module in the aforementioned data storage device may be implemented in whole or in part through software, hardware, or a combination thereof. Each module may be embedded in or independent of a processor in a computer device in the form of hardware, or may be stored in a memory in the computer device in the form of software, so that the processor can call and execute the corresponding operations of each module.

[0122] In an exemplary embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as shown in FIG. Figure 7As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, abbreviated as I / O) and a communication interface. The processor, memory and input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The database of the computer device is used to store key-value data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a data storage method is implemented.

[0123] Those skilled in the art will understand that Figure 7 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0124] In an exemplary embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and the processor implements the steps of the above-mentioned data storage method when executing the computer program.

[0125] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned data storage method are implemented.

[0126] In one embodiment, a computer program product is provided, comprising a computer program, which implements the steps of the above-mentioned data storage method when executed by a processor.

[0127] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0128] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the above-mentioned embodiments. In particular, any reference to memory, database, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, distributed databases based on blockchains. The processor involved in the various embodiments provided herein may be, but are not limited to, a general-purpose processor, a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), a programmable logic unit (PLC), a data processing logic unit based on quantum computing, an artificial intelligence (AI) processor, and the like.

[0129] The technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0130] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present application. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

Claims

1. A data storage method, characterized in that: The method comprises: Get the data to be stored and the first address offset of the memory page; determining index information of the data according to the first address offset of the memory page; storing the index information and the data in a data block; wherein the data block is stored in the memory page; The memory page is sent to a storage thread, and the storage thread is used to store the data according to the index information stored in the data block in the memory page.

2. The method according to claim 1, characterized in that The steps of obtaining the first address offset of a memory page include: If the memory page is the first memory page among multiple memory pages, a first address offset of the memory page is determined according to a memory base address.

3. The method according to claim 1, characterized in that The steps of obtaining the first address offset of a memory page include: If the memory page is not the first memory page among the multiple memory pages, the first address offset of the memory page is determined according to a message header in a previous memory page corresponding to the memory page.

4. The method according to claim 1, wherein The determining the index information of the data according to the first address offset of the memory page includes: Obtaining the data length of the data; Determining a second address offset corresponding to the data according to the first address offset of the memory page and the data length; The index information of the data is obtained according to the second address offset and the data length.

5. The method according to claim 4, characterized in that The data block includes an index area and a data area; Storing the index information and the data in a data block includes: storing the second address offset and the data length in the index information of the data in the index area in the data block; storing the data in the data area of ​​the data block; The data block storing the index information and the data is obtained according to the index area including the second address offset and the data length, and the data area including the data.

6. The method according to claim 5, characterized in that Storing the data according to the index information stored in the data block in the memory page includes: For each of the data blocks in the memory page, determining a second address offset and a data length corresponding to the index area in the data block; According to the second address offset and the data length, the corresponding data is obtained from the data area of ​​the data block for storage.

7. The method according to claim 6, characterized in that The data block stores an offset record; the offset record represents the third address offset of the next data block of the data block; After acquiring the corresponding data from the data area of ​​the data block and storing the data according to the second address offset and the data length, the method further includes: Determining a third address offset of the next data block according to the offset record in the data block; Obtaining the next data block according to the third address offset; According to the index information stored in the next data block, the data in the next data block is stored, and the step of determining the third address offset of the next data block according to the offset record in the data block is returned to execute until the data in each data block in the memory page is stored.

8. A data storage device, characterized in that The device comprises: An acquisition module, configured to acquire data to be stored and a first address offset of a memory page; a determination module, configured to determine index information of the data according to the first address offset of the memory page; A first storage module is configured to store the index information and the data in a data block; the data block is stored in the memory page; The second storage module is used to send the memory page to a storage thread, and the storage thread is used to store the data according to the index information stored in the data block in the memory page.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

11. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images