Data writing and reading method, device and data reading and writing system

Abstract

The present application provides a data writing and reading method, device and data reading and writing system, wherein the data writing method is applied to a storage server; the method comprises: receiving a data writing request from a user; wherein the data writing request carries a target key-value pair; the target key-value pair comprises a target key and a target key value; if the byte length of the target key value is greater than a preset length threshold, writing the target key value to a designated file of a target storage engine of the storage server; generating a new target key value corresponding to the target key value based on the position of the target key value in the designated file; and writing the new target key value and the target key as a new target key-value pair to the target storage engine. The present application can alleviate the problem of consuming system performance caused by the engine write amplification problem in the prior art, and on the other hand, when performing a data reading operation, it can reduce the amount of data scanned and eliminate potential system performance bottlenecks.

Description

Technical Field

[0001] The present application relates to the field of software testing technology, and in particular to a data writing and reading method, device, and data reading and writing system. Background Art

[0002] Distributed Table Storage is a fully managed distributed key-value database service that primarily stores user data in the form of tables. User-created tables are stored in a specific order and horizontally partitioned into multiple shards. Shard data is stored on storage servers, and the corresponding metadata (shard key range and corresponding storage server) is stored on the management server. Proxy servers provide external services.

[0003] Sharded data stored on storage servers generally uses a single-shard, single-engine approach, such as the commonly used stand-alone databases like LevelDB and RocksDB. These engines generally append data to the log before writing it to memory. Using a specific strategy, they periodically persist the data in memory to media such as disk. Furthermore, to improve read efficiency, the persisted data needs to be compressed and merged. As a result, the engine may write multiple times for a single user write request, creating a write amplification problem. On the other hand, when reading data based on a user read request, a full-engine-level traversal operation is required, scanning the data from beginning to end to obtain the desired data.

[0004] In summary, for key-value data with a large byte length, due to the engine's write amplification problem, a large amount of data needs to be written, which can easily lead to system performance degradation; and when the system reads data, a large amount of data needs to be scanned, which can easily cause system bottlenecks. Summary of the Invention

[0005] The purpose of this application is to provide a data writing and reading method, device and data reading and writing system, which can alleviate the problem of consuming system performance caused by the engine write amplification problem in the prior art. On the other hand, when performing a read operation, it can reduce the scanned data and eliminate potential system performance bottlenecks.

[0006] In a first aspect, an embodiment of the present application provides a data writing method, which is applied to a storage server; the method includes: receiving a data writing request from a user; wherein the data writing request carries a target key-value pair; the target key-value pair includes a target key and a target key value; if the byte length of the target key value is greater than a preset length threshold, writing the target key value to a designated file of a target storage engine of the storage server; the designated file is a readable and writable file externally bound to the target storage engine; generating a new target key value corresponding to the target key value based on the position of the target key value in the designated file; and writing the new target key value and the target key as a new target key-value pair to the target storage engine.

[0007] Furthermore, the above-mentioned step of generating a new target key value corresponding to the target key value based on the position of the target key value in the specified file includes: obtaining the position offset value and byte length of the target key value in the specified file; and generating a new target key value corresponding to the target key value according to the position offset value and byte length.

[0008] Furthermore, the above method also includes: if the byte length of the target key value is less than a preset length threshold, writing the target key-value pair into the target storage engine.

[0009] Furthermore, the above-mentioned storage server stores key intervals corresponding to multiple data shards; each data shard corresponds to a storage engine; before the step of writing the target key value into the designated file of the target storage engine of the storage server, it also includes: searching for the target data shard corresponding to the target key according to the key intervals corresponding to the multiple data shards; and using the storage engine corresponding to the target data shard as the target storage engine.

[0010] Furthermore, the above-mentioned step of searching for the target data shard corresponding to the target key according to the key intervals corresponding to multiple data shards includes: searching for the target key interval to which the target key belongs according to the key intervals corresponding to multiple data shards; and determining the data shard corresponding to the target key interval as the target data shard corresponding to the target key.

[0011] Furthermore, the above method also includes: assigning a label to the new target key value.

[0012] In the second aspect, an embodiment of the present application also provides a data reading method, which is applied to a storage server, and the data written by the method described in the first aspect is stored in a designated file of the storage server, and the method includes: receiving a data reading request from a user; wherein the data reading request carries a target key; searching for a target key value corresponding to the target key through the target storage engine of the storage server; if the target key value points to a designated file, obtaining the data corresponding to the data reading request from the designated file according to the target key value; wherein the designated file is a readable and writable file externally bound to the target storage engine.

[0013] Furthermore, the step of obtaining data corresponding to the data reading request from the designated file according to the target key value includes: using the target key value as index data to search for data corresponding to the data reading request from the designated file.

[0014] Furthermore, the above method also includes: if the target key value does not point to the designated file, determining the target key value as the data corresponding to the data read request.

[0015] Furthermore, the above-mentioned storage server stores key intervals corresponding to multiple data shards; each data shard corresponds to a storage engine; before the step of searching for the target key value corresponding to the target key through the target storage engine of the storage server, it also includes: searching for the target data shard corresponding to the target key according to the key intervals corresponding to the multiple data shards; and using the storage engine corresponding to the target data shard as the target storage engine.

[0016] Furthermore, the above method also includes: detecting whether the target key value is marked with a specified tag; if so, determining that the target key value points to the specified file; if not, determining that the target key value does not point to the specified file.

[0017] In a third aspect, an embodiment of the present application also provides a data writing device, which is applied to a storage server; the method includes: a write request receiving module, used to receive a user's data writing request; wherein the data writing request carries a target key-value pair; the target key-value pair includes a target key and a target key value; a key value writing module, used to write the target key value to a designated file of a target storage engine of the storage server if the byte length of the target key value is greater than a preset length threshold; the designated file is a readable and writable file externally bound to the target storage engine; a new key value generation module, used to generate a new target key value corresponding to the target key value based on the position of the target key value in the designated file; a key value pair writing module, used to write the new target key value and the target key as a new target key value pair to the target storage engine.

[0018] In the fourth aspect, an embodiment of the present application also provides a data reading device, which is applied to a storage server, and the data written by the method described in the first aspect is stored in a designated file of the storage server. The device includes: a read request receiving module, which is used to receive a user's data reading request; wherein the data reading request carries a target key; a key value search module, which is used to search for a target key value corresponding to the target key through the target storage engine of the storage server; a data reading module, which is used to obtain data corresponding to the data reading request from the designated file according to the target key value if the target key value points to a designated file; wherein the designated file is a readable and writable file externally bound to the target storage engine.

[0019] In the fifth aspect, an embodiment of the present application also provides a data reading and writing system, the system including: a proxy server and a storage server; the proxy server is communicatively connected to the storage server; the proxy server is used to send the user's data writing request or data reading request to the storage server; the storage server is used to execute the method described in the first aspect or the second aspect.

[0020] In a sixth aspect, an embodiment of the present application further provides an electronic device comprising a processor and a memory, wherein the memory stores computer-executable instructions that can be executed by the processor, and the processor executes the computer-executable instructions to implement the method described in the first or second aspect above.

[0021] In the seventh aspect, an embodiment of the present application also provides a computer-readable storage medium, which stores computer-executable instructions. When the computer-executable instructions are called and executed by a processor, the computer-executable instructions prompt the processor to implement the method described in the first or second aspect above.

[0022] The data writing method provided in the embodiment of the present application is applied to a storage server; when a user's data writing request is received, a judgment is made based on the target key value in the target key-value pair carried in the request; if the byte length of the target key value is greater than a preset length threshold, the target key value is written to a designated file of a target storage engine of the storage server; the designated file is a readable and writable file externally bound to the target storage engine; and a new target key value corresponding to the target key value is generated based on the position of the target key value in the designated file; the new target key value and the target key are written to the target storage engine as a new target key-value pair.

[0023] This method can perform special processing on target key values ​​with larger byte lengths, separate the target key value from the target key, write the target key value into the specified file of the target storage engine, that is, the readable and writable file bound outside the engine, and write the target key and the new target key value generated based on the position of the target key value in the specified file into the target storage engine, so that the data written into the target storage engine is relatively small, alleviating the problem of consuming system performance caused by the engine write amplification problem in the prior art. On the other hand, the data written in this way can reduce the amount of data scanned by the system when performing data reading operations, eliminating potential system performance bottlenecks. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the specific implementation methods of the present application or the technical solutions in the prior art, the following is a brief introduction to the drawings required for use in the specific implementation methods or the description of the prior art. Obviously, the drawings described below are some implementation methods of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0025] Figure 1 A flowchart of a data writing method provided in an embodiment of the present application;

[0026] Figure 2 A flowchart of a data reading method provided in an embodiment of the present application;

[0027] Figure 3 A structural block diagram of a data writing device provided in an embodiment of the present application;

[0028] Figure 4 A structural block diagram of a data reading device provided in an embodiment of the present application;

[0029] Figure 5 A structural block diagram of a data reading and writing system provided in an embodiment of the present application;

[0030] Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0031] The following will clearly and completely describe the technical solutions of this application in conjunction with the embodiments. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0032] Currently, for key-value data with a large byte length, due to the engine's write amplification problem, a large amount of data needs to be written, which can easily lead to a decline in system performance; and when the system reads data, a large amount of data needs to be scanned, which can easily cause system bottlenecks.

[0033] Based on this, the embodiments of the present application provide a data writing and reading method, device and data reading and writing system, which can alleviate the problem of consuming system performance caused by the engine write amplification problem in the prior art. On the other hand, when performing data reading operations, the scanned data is reduced, which can eliminate potential system performance bottlenecks.

[0034] Figure 1 This is a flowchart of a data writing method provided in an embodiment of the present application, which is applied to a storage server; the data writing method specifically includes the following steps:

[0035] Step S102: Receive a data write request from a user; wherein the data write request carries a target key-value pair; the target key-value pair includes a target key and a target key value.

[0036] In a distributed table storage service, the above-mentioned data write request is usually forwarded by the proxy server to the storage server. The proxy server needs to determine which storage server to forward it to. The proxy server pre-caches the key interval corresponding to each data shard and the corresponding relationship with the storage server. Therefore, after receiving the user's data write request, the proxy server can find the corresponding data shard according to the target key carried in the request, thereby determining the target storage server and forwarding the above-mentioned user's data write request to it.

[0037] Step S104: If the byte length of the target key value is greater than a preset length threshold, the target key value is written into a designated file of the target storage engine of the storage server.

[0038] The above data write request carries a target key-value pair, that is, the data that the user wants to store. The target key-value pair includes a target key and a target key value. After obtaining the target key value, the storage server needs to first make a judgment based on its byte length and the preset length threshold. If the byte length of the target key value is greater than the preset length threshold, the target key value is written to the designated file of the target storage engine of the storage server. The designated file is a readable and writable file bound to the outside of the target storage engine, and is two different storage areas from the inside of the target storage engine.

[0039] It should be noted that a storage server can be configured with one or more storage engines, each of which can be bound to one or more designated files for storing key values ​​with larger byte lengths. There are various ways to determine the target storage engine, which are not specifically limited here.

[0040] Step S106: Generate a new target key value corresponding to the target key value based on the position of the target key value in the designated file.

[0041] The position of the target key value in the specified file may include the position offset value and byte length of the target key value in the specified file. These two data can be used to determine the specific position of the target key value in the specified file. When the target key value is written to the specified file of the target storage engine of the storage server, a new target key value corresponding to the target key value needs to be generated based on the position of the target key value in the specified file. In other words, the position offset value and byte length of the target key value in the specified file can be used as the new target key value.

[0042] Step S108: Write the new target key value and the target key as a new target key value pair into the target storage engine.

[0043] After the new target key value is determined, the target key value and the target key carried in the data write request are further written into the target storage engine as a new target key-value pair.

[0044] The data writing method provided in the embodiment of the present application can perform special processing on target key values ​​with larger byte lengths, that is, separating the target key value from the target key, writing the target key value into the specified file of the target storage engine, and writing the target key and the new target key value generated based on the position of the target key value in the specified file into the target storage engine, thereby making the data written in the target storage engine relatively small, alleviating the problem of consuming system performance caused by the engine write amplification problem in the prior art. On the other hand, the data written in this way can reduce the amount of data scanned by the system when performing data reading operations, eliminating potential system performance bottlenecks.

[0045] In order to improve the accuracy of data search, in an embodiment of the present application, the step of generating a new target key value corresponding to the target key value based on the position of the target key value in the specified file can be implemented in the following way:

[0046] Get the position offset value and byte length of the target key value in the specified file; generate a new target key value corresponding to the target key value based on the position offset value and byte length.

[0047] In the distributed table storage service, the storage server also stores the key intervals corresponding to multiple data shards. Each data shard corresponds to a storage engine. For example, data shard A corresponds to the key interval 1-100 and the storage engine a. Data shard B corresponds to the key interval 100-200 and the storage engine b. Data shard C corresponds to the key interval 200-300 and the storage engine c, and so on.

[0048] Before the step of writing the target key value into the designated file of the target storage engine of the storage server, the target storage engine is determined. For details, see the following method:

[0049] (1) According to the key intervals corresponding to multiple data shards, find the target data shard corresponding to the target key.

[0050] Specifically, according to the key intervals corresponding to the multiple data shards, the target key interval to which the target key belongs is searched; for example, if the target key is 30, then it belongs to the target key interval 1-100.

[0051] The data shard corresponding to the target key range is determined as the target data shard corresponding to the target key. If the data shard corresponding to the target key range 1-100 is A, then data shard A is used as the target data shard.

[0052] (2) The storage engine corresponding to the target data shard is used as the target storage engine. If the storage engine corresponding to data shard A is a, then storage engine a is the target storage engine.

[0053] In order to facilitate the storage server to easily determine whether the target key value found is a target key value with a byte length less than a threshold value, or information used to represent the position of the target key value with a byte length greater than the threshold value in the specified file when reading data, the embodiment of the present application also provides a preferred implementation method, after the new target key value is determined as described above, a specified tag is marked for the new target key value. Through this specified tag, it can be determined that when reading data, the target key value found is information used to represent the position of the target key value with a byte length greater than the threshold value in the specified file, and then the corresponding data can be found from the specified file based on the above information.

[0054] The above describes how to write target key values ​​whose byte length exceeds the threshold. For target key values ​​whose byte length does not exceed the threshold, you can still use the conventional data writing method, that is, write the target key-value pair directly to the target storage engine.

[0055] The embodiment of the present application classifies and stores key values ​​of different byte lengths. For key values ​​whose byte length is greater than a threshold, the key value and key are separated. Only the position information and key corresponding to the key value in the specified file are written into the engine, and the key value is written into the specified file of the engine. In this way, the engine data will not be too large, thereby alleviating the problem of consuming system performance caused by the engine write amplification problem in the prior art. On the other hand, the data written in this way can reduce the amount of data scanned by the system when performing data reading operations, eliminating potential system performance bottlenecks.

[0056] Based on the above data writing method, the embodiment of the present application also provides a data reading method, which is applied to a storage server, wherein the designated file of the storage server stores the data written by the data writing method of the above embodiment, see Figure 2 The flowchart of a data reading method provided in an embodiment of the present application is shown, and the method specifically includes the following steps:

[0057] Step S202: Receive a data reading request from a user; wherein the data reading request carries a target key.

[0058] Similar to the data write request above, the user's data read request can also be forwarded by the proxy server. This data read request only carries the target key.

[0059] Step S204: searching for a target key value corresponding to the target key through the target storage engine of the storage server.

[0060] A storage server can include one or more storage engines. There are various ways to determine the target storage engine, which are not specifically limited here. The target key value corresponding to the target key can be found through the target storage engine. For example, calling the data read interface of the target storage engine can obtain the target key value corresponding to the target key.

[0061] Step S206: If the target key value points to a designated file, data corresponding to the data reading request is obtained from the designated file according to the target key value.

[0062] The specified file is a readable and writable file externally bound to the target storage engine. There are various ways to determine whether the target key value points to the specified file. For example, you can check whether the target key value has a specified tag, and if so, point to the file; otherwise, do not. Alternatively, you can determine based on the byte length: if the byte length is less than a second threshold, point to the file; otherwise, do not. Alternatively, you can traverse the specified file each time to see if there is an index, and if so, point to the file; otherwise, do not.

[0063] If the target key value points to a specified file, the data corresponding to the data reading request is obtained from the specified file according to the target key value. In this case, the target key value corresponds to the location information of the original target key value in the specified file, so that the corresponding original target key value data in the specified file can be found according to the location information.

[0064] The above step of obtaining the data corresponding to the data read request from the specified file according to the target key value can be implemented in the following way: using the target key value as index data, searching for the data corresponding to the data read request from the specified file.

[0065] If the storage server stores key intervals corresponding to multiple data shards and each data shard corresponds to a storage engine, the target storage engine needs to be determined as follows;

[0066] According to the key intervals corresponding to the multiple data shards, the target data shard corresponding to the target key is searched; the storage engine corresponding to the target data shard is used as the target storage engine. The specific implementation method is the same as that in the previous embodiment and will not be repeated here.

[0067] The above is an explanation of the case where the target key value points to a specified file. If the target key value does not point to a specified file, the target key value can be directly determined as the data corresponding to the data read request.

[0068] The data reading method provided in the embodiment of the present application is a data reading operation performed on the data written to the storage server by the data writing method provided in the previous embodiment. For key values ​​whose byte length exceeds a threshold, it is necessary to search for the specific data in the specified file. During this data reading process, since the storage engine stores relatively little data, it is not necessary to scan a large amount of data, which can avoid causing system performance bottlenecks.

[0069] In view of the above data writing method, the embodiment of the present application further provides a data writing device, which is applied to a storage server; see Figure 3 As shown, the data writing device includes:

[0070] A write request receiving module 302 is used to receive a user's data write request; wherein the data write request carries a target key-value pair; the target key-value pair includes a target key and a target key value; a key value writing module 304 is used to write the target key value to a designated file of the target storage engine of the storage server if the byte length of the target key value is greater than a preset length threshold; the designated file is a readable and writable file externally bound to the target storage engine; a new key value generating module 306 is used to generate a new target key value corresponding to the target key value based on the position of the target key value in the designated file; a key value pair writing module 308 is used to write the new target key value and the target key as a new target key value pair to the target storage engine.

[0071] In another possible implementation, the new key value generation module 306 is further configured to: obtain a position offset value and a byte length of the target key value in a specified file; and generate a new target key value corresponding to the target key value according to the position offset value and the byte length.

[0072] In another possible implementation, the key-value pair writing module 308 is further configured to write the target key-value pair into the target storage engine if the byte length of the target key value is less than a preset length threshold.

[0073] In another possible embodiment, the storage server stores key intervals corresponding to multiple data shards; each data shard corresponds to a storage engine; the above-mentioned data writing device also includes a target storage engine determination module, which is used to search for the target data shard corresponding to the target key based on the key intervals corresponding to the multiple data shards; and the storage engine corresponding to the target data shard is used as the target storage engine.

[0074] In another possible implementation, the target storage engine determination module is further used to: search for the target key interval to which the target key belongs based on the key intervals corresponding to multiple data shards; and determine the data shard corresponding to the target key interval as the target data shard corresponding to the target key.

[0075] In another possible implementation, the data writing device further includes: a label marking module, configured to mark a new target key value with a specified label.

[0076] The data writing device provided in the embodiment of the present application has the same implementation principle and technical effects as those in the aforementioned data writing method embodiment. For the sake of brief description, for matters not mentioned in the embodiment of the data writing device, reference may be made to the corresponding content in the aforementioned data writing method embodiment.

[0077] In view of the above data reading method, the embodiment of the present application further provides a data reading device, which is applied to a storage server. The designated file of the storage server stores the data written by the method of the above data writing method embodiment, see Figure 4 As shown, the data reading device includes:

[0078] The read request receiving module 402 is used to receive the user's data read request; wherein the data read request carries a target key; the key value search module 404 is used to search for the target key value corresponding to the target key through the target storage engine of the storage server; the data reading module 406 is used to obtain the data corresponding to the data read request from the specified file according to the target key value if the target key value points to the specified file; wherein the specified file is a readable and writable file externally bound to the target storage engine.

[0079] In another possible implementation, the data reading module 406 is further configured to use the target key value as index data to search for data corresponding to the data reading request from a specified file.

[0080] In another possible implementation, the data reading module 406 is further configured to: if the target key value does not point to a designated file, determine the target key value as data corresponding to the data reading request.

[0081] In another possible embodiment, the storage server stores key intervals corresponding to multiple data shards; each data shard corresponds to a storage engine; the above-mentioned data reading device also includes: a target storage engine determination module, which is used to search for the target data shard corresponding to the target key based on the key intervals corresponding to the multiple data shards; and use the storage engine corresponding to the target data shard as the target storage engine.

[0082] In another possible implementation, the apparatus further includes a determination module configured to detect whether the target key value is marked with a specified tag; if so, determine that the target key value points to a specified file; if not, determine that the target key value does not point to a specified file.

[0083] The data reading device provided in the embodiment of the present application has the same implementation principle and technical effects as those of the aforementioned data reading method embodiment. For the sake of brief description, for matters not mentioned in the embodiment of the data reading device, reference may be made to the corresponding content in the aforementioned data reading method embodiment.

[0084] Based on the above data writing method and data reading method, the present application embodiment also provides a data reading and writing system, see Figure 5 As shown, the system includes: a proxy server 52 and a storage server 54; the proxy server 52 is communicatively connected to the storage server 54; the proxy server 52 is used to send the user's data write request or data read request to the storage server 54; the storage server 54 is used to execute the steps of the above-mentioned data writing method or the above-mentioned data reading method.

[0085] The data reading and writing system provided in the embodiment of the present application has the same implementation principle and technical effects as those of the aforementioned method embodiment. For the sake of brief description, for matters not mentioned in the embodiment of the data reading and writing system, reference may be made to the corresponding content in the aforementioned method embodiment.

[0086] The present application also provides an electronic device, such as Figure 6 As shown, it is a structural diagram of the electronic device, wherein the electronic device includes a processor 61 and a memory 60, the memory 60 stores computer executable instructions that can be executed by the processor 61, and the processor 61 executes the computer executable instructions to implement the above method.

[0087] exist Figure 6 In the illustrated embodiment, the electronic device further includes a bus 62 and a communication interface 63 , wherein the processor 61 , the communication interface 63 and the memory 60 are connected via the bus 62 .

[0088] Among them, the memory 60 may include a high-speed random access memory (RAM), and may also include a non-volatile memory (non-volatile memory), such as at least one disk storage. The communication connection between the system network element and at least one other network element is realized through at least one communication interface 63 (which can be wired or wireless), and the Internet, wide area network, local area network, metropolitan area network, etc. can be used. The bus 62 can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. The bus 62 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 6 Only one bidirectional arrow is used in the diagram, but this does not mean that there is only one bus or one type of bus.

[0089] The processor 61 may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by hardware integrated logic circuits in the processor 61 or by software instructions. The processor 61 may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in the embodiments of the present application may be directly implemented and executed by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. The software module may be located in a storage medium mature in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, etc. The storage medium is located in the memory, and the processor 61 reads the information in the memory and completes the steps of the method of the above embodiment in combination with its hardware.

[0090] An embodiment of the present application also provides a computer-readable storage medium, which stores computer-executable instructions. When the computer-executable instructions are called and executed by the processor, the computer-executable instructions prompt the processor to implement the above-mentioned method. The specific implementation can be found in the above-mentioned method embodiment, which will not be repeated here.

[0091] The computer program products of the data writing and reading methods, devices, and data reading and writing systems provided in the embodiments of the present application include a computer-readable storage medium storing program code. The instructions included in the program code can be used to execute the methods described in the previous method embodiments. For specific implementation, please refer to the method embodiments and will not be repeated here.

[0092] Unless otherwise specifically stated, the relative steps, numerical expressions and values ​​of the components and steps set forth in these embodiments do not limit the scope of the present application.

[0093] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium that is executable by a processor. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0094] In the description of this application, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicating orientations or positional relationships, are based on the orientations or positional relationships shown in the accompanying drawings and are intended solely to facilitate the description of this application and simplify the description. They do not indicate or imply that the devices or components referred to must have a specific orientation, be constructed, or operate in a specific orientation. Therefore, they should not be construed as limitations on this application. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0095] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present application, which are used to illustrate the technical solutions of the present application, rather than to limit them. The scope of protection of the present application is not limited thereto. Although the present application has been described in detail with reference to the above-described embodiments, those skilled in the art should understand that any person skilled in the art can modify or easily conceive of changes to the technical solutions described in the above-described embodiments within the technical scope disclosed in the present application, or perform equivalent replacements for some of the technical features thereof. These modifications, changes, or replacements do not deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application shall be subject to the scope of protection of the claims.

Claims

1. A data writing method, characterized in that: The method is applied to a storage server in a distributed table storage service scenario; The storage server stores key intervals corresponding to a plurality of data shards respectively; Each data shard corresponds to a storage engine; the method includes: Receive a data write request from a user; wherein the data write request carries a target key-value pair; the target key-value pair includes a target key and a target key value; Searching for a target data shard corresponding to the target key according to the key intervals corresponding to the plurality of data shards; and using the storage engine corresponding to the target data shard as the target storage engine; If the byte length of the target key value is greater than a preset length threshold, the target key value is written into a designated file of the target storage engine of the storage server; the designated file is a readable and writable file externally bound to the target storage engine; Generate a new target key value corresponding to the target key value based on the position of the target key value in the specified file; The new target key value and the target key are written into the target storage engine as a new target key-value pair.

2. The method according to claim 1, characterized in that The step of generating a new target key value corresponding to the target key value based on the position of the target key value in the designated file includes: Obtain the position offset value and byte length of the target key value in the specified file; Generate a new target key value corresponding to the target key value according to the position offset value and the byte length.

3. The method according to claim 1, characterized in that The method further comprises: If the byte length of the target key value is less than a preset length threshold, the target key-value pair is written into the target storage engine.

4. The method according to claim 1, wherein The step of searching for a target data shard corresponding to the target key according to key intervals corresponding to the plurality of data shards includes: According to the key intervals corresponding to the multiple data shards, the target key interval to which the target key belongs is searched; The data shard corresponding to the target key interval is determined as the target data shard corresponding to the target key.

5. The method according to claim 1, characterized in that The method further comprises: Specifies the label for the new target key.

6. A data reading method, characterized in that: The method is applied to a storage server in a distributed table storage service scenario, wherein the storage server stores key intervals corresponding to multiple data shards; each of the data shards corresponds to a storage engine; The designated file of the storage server stores data written by the method according to any one of claims 1 to 5, the method comprising: Receive a data read request from a user; wherein the data read request carries a target key; Searching for a target data shard corresponding to the target key according to the key intervals corresponding to the plurality of data shards; and using the storage engine corresponding to the target data shard as the target storage engine; Searching for a target key value corresponding to the target key through a target storage engine of the storage server; If the target key value points to a designated file, the data corresponding to the data read request is obtained from the designated file according to the target key value; wherein the designated file is a readable and writable file externally bound to the target storage engine.

7. The method according to claim 6, characterized in that The step of obtaining data corresponding to the data read request from the designated file according to the target key value includes: The target key value is used as index data to search for data corresponding to the data read request from the specified file.

8. The method according to claim 6, characterized in that The method further comprises: If the target key value does not point to the designated file, the target key value is determined as the data corresponding to the data read request.

9. The method according to claim 6, characterized in that The method further comprises: Detect whether the target key value is marked with a specified tag; If yes, determining that the target key value points to the specified file; If not, it is determined that the target key value does not point to the designated file.

10. A data writing device, characterized in that: The device is applied to a storage server in a distributed table storage service scenario; The storage server stores key intervals corresponding to a plurality of data shards respectively; Each of the data shards corresponds to a storage engine; the device includes: A write request receiving module, configured to receive a data write request from a user; wherein the data write request carries a target key-value pair; the target key-value pair includes a target key and a target key value; An engine determination module is configured to search for a target data shard corresponding to the target key according to key intervals corresponding to the plurality of data shards; and use the storage engine corresponding to the target data shard as the target storage engine; A key value writing module is configured to write the target key value into a designated file of a target storage engine of the storage server if the byte length of the target key value is greater than a preset length threshold; the designated file is a readable and writable file externally bound to the target storage engine; A new key value generating module, configured to generate a new target key value corresponding to the target key value based on the position of the target key value in the specified file; A key-value pair writing module is used to write the new target key value and the target key into the target storage engine as a new target key-value pair.

11. A data reading device, characterized in that: The device is applied to a storage server in a distributed table storage service scenario, wherein the storage server stores key intervals corresponding to a plurality of data shards; each of the data shards corresponds to a storage engine; The designated file of the storage server stores data written by the method according to any one of claims 1 to 6, and the device includes: A read request receiving module, configured to receive a data read request from a user; wherein the data read request carries a target key; An engine determination module is configured to search for a target data shard corresponding to the target key according to key intervals corresponding to the plurality of data shards; and use the storage engine corresponding to the target data shard as the target storage engine; A key value search module, configured to search for a target key value corresponding to the target key through a target storage engine of the storage server; A data reading module is used to obtain the data corresponding to the data reading request from the specified file according to the target key value if the target key value points to the specified file; wherein, the specified file is a readable and writable file externally bound to the target storage engine.

12. A data reading and writing system, characterized in that: The system includes: a proxy server and a storage server in a distributed table storage service; The proxy server is communicatively connected with the storage server; The proxy server is used to send the user's data writing request or data reading request to the storage server; The storage server is used to execute the steps of the data writing method according to any one of claims 1 to 5 or the data reading method according to any one of claims 6 to 9.

13. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores computer-executable instructions that can be executed by the processor, and the processor executes the computer-executable instructions to implement the method according to any one of claims 1 to 5 or 6 to 9.

14. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer-executable instructions. When the computer-executable instructions are called and executed by a processor, the computer-executable instructions prompt the processor to implement the method according to any one of claims 1 to 5 or 6 to 9.

Images