Data storage method, data reading method, device, equipment and medium

Abstract

Embodiments of the present disclosure relate to a data storage method, a data reading method, an apparatus, a device and a medium. The data storage method comprises: in response to a storage instruction for target data, obtaining the size of the target data; if the size of the target data is greater than a preset compression threshold, performing compression processing on the target data to obtain compressed data and compression processing information; if the size of the compressed data is greater than a preset splitting threshold, performing splitting processing on the compressed data to obtain a plurality of split data and splitting processing information; and based on a key index of the target data, storing the plurality of split data in a target storage system in association. The present disclosure can enable the data to be stored in the storage system in the form of small data, effectively reduce the data reading time, and help further reduce the service response time and ensure service stability.

Description

Technical Field

[0001] This disclosure relates to the field of data processing technology, and in particular to a data storage method, a data reading method, an apparatus, a device, and a medium. Background Technology

[0002] In most fields today, large amounts of data are involved, requiring data storage and retrieval operations. Current technologies mostly store data directly, but in many cases, the data is very large, and retrieving large amounts of data often takes a long time, resulting in long service response times and potentially negatively impacting service stability. Summary of the Invention

[0003] In order to solve the above-mentioned technical problems, or at least partially solve the above-mentioned technical problems, this disclosure provides a data storage method, a data reading method, an apparatus, a device, and a medium.

[0004] In a first aspect, embodiments of this disclosure provide a data storage method, the method comprising: in response to a storage instruction for target data, obtaining the size of the target data; the storage instruction being used to instruct the target data to be stored in a target storage system; if the size of the target data is greater than a preset compression threshold, compressing the target data to obtain compressed data and compression processing information; if the size of the compressed data is greater than a preset splitting threshold, splitting the compressed data to obtain multiple split data and splitting processing information; and storing the multiple split data in association in the target storage system based on a key index of the target data; wherein the split data associated with the key index of the target data in the target storage system carries the splitting processing information and compression processing information of the target data.

[0005] Secondly, embodiments of this disclosure provide a data reading method, the method comprising: responding to a read instruction for target data, searching for associated data corresponding to a key index of the target data in a target storage system; the read instruction being used to instruct the reading of the target data from the target storage system; if the associated data is found and the associated data contains splitting processing information and compression processing information of the target data, obtaining multiple split data obtained by splitting the target data from the target storage system based on the splitting processing information and the key index of the target data; merging the obtained multiple split data to obtain merged data; decompressing the merged data according to the compression processing information, and using the decompressed data as the read target data.

[0006] Thirdly, embodiments of this disclosure provide a data storage device, the device comprising: a size acquisition module, configured to acquire the size of the target data in response to a storage instruction for the target data; the storage instruction being configured to instruct the target data to be stored in a target storage system; a compression processing module, configured to compress the target data if the size of the target data is greater than a preset compression threshold, to obtain compressed data and compression processing information; a splitting processing module, configured to split the compressed data if the size of the compressed data is greater than a preset splitting threshold, to obtain multiple split data and splitting processing information; and an associated storage module, configured to associate and store the multiple split data in the target storage system based on a key index of the target data.

[0007] Fourthly, embodiments of this disclosure provide a data reading device, the device comprising: an associated data lookup module, configured to, in response to a read instruction for target data, look up associated data corresponding to a key index of the target data in a target storage system; the read instruction instructs the reading of the target data from the target storage system; a split data acquisition module, configured to, if the associated data is found and the associated data packet contains split processing information and compression processing information of the target data, acquire multiple split data obtained by splitting the target data from the target storage system based on the split processing information and the key index of the target data; a merging processing module, configured to merge the acquired multiple split data to obtain merged data; and a decompression processing module, configured to decompress the merged data according to the compression processing information, and use the decompressed data as the read target data.

[0008] Fifthly, embodiments of this disclosure provide an electronic device, the electronic device comprising: a processor; a memory for storing executable instructions of the processor; the processor being configured to read the executable instructions from the memory and execute the instructions to implement the data storage method or data reading method provided in embodiments of this disclosure.

[0009] Sixthly, embodiments of this disclosure also provide a computer-readable storage medium storing a computer program for executing the data storage method or data reading method provided in embodiments of this disclosure.

[0010] The data storage method and apparatus provided in this disclosure, when storing target data, can set compression thresholds and splitting thresholds. First, it can determine whether the target data needs to be compressed based on the compression threshold. If compression is required (when the size of the target data is greater than the compression threshold), after obtaining the compressed data, it will further determine whether the compressed data needs to be split based on the splitting threshold. If splitting is required (when the size of the split data is greater than the splitting threshold), the compressed data is split, and the resulting multiple split data are stored together in the target storage system. This method can reasonably and effectively process large amounts of data, allowing the data to be stored in the target storage system in smaller data formats, which is more conducive to subsequent data reading operations. Furthermore, the split data associated with the key index of the target data carries the splitting and compression processing information of the target data, facilitating accurate reconstruction of the required target data. This data storage method can effectively improve subsequent data reading efficiency, reduce data reading time, help further reduce service response time, and ensure service stability.

[0011] The data reading method and apparatus provided in this disclosure, when reading target data, can first search for associated data corresponding to the key index of the target data in the target storage system. If associated data is found and the associated data packet contains splitting and compression information of the target data, multiple split data obtained by splitting the target data can be obtained from the target storage system based on the splitting information and the key index of the target data. Then, the multiple split data can be further merged to obtain merged data. Finally, the merged data is decompressed according to the compression information, and the decompressed data is used as the read target data. This method can directly read and merge smaller split data, which, compared with directly reading larger data, can effectively improve data reading efficiency, reduce data reading time, and help further reduce service response time and ensure service stability.

[0012] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0013] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0014] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0015] Figure 1 A schematic flowchart illustrating a data storage method provided in an embodiment of this disclosure;

[0016] Figure 2 A schematic flowchart illustrating a data storage method provided in an embodiment of this disclosure;

[0017] Figure 3 A flowchart illustrating a data reading method provided in an embodiment of this disclosure;

[0018] Figure 4 A flowchart illustrating a data reading method provided in an embodiment of this disclosure;

[0019] Figure 5 This is a schematic diagram illustrating successful data processing according to an embodiment of the present disclosure;

[0020] Figure 6 This is a schematic diagram illustrating a data processing failure as provided in an embodiment of the present disclosure.

[0021] Figure 7 This is a schematic diagram of the structure of a data storage device provided in an embodiment of the present disclosure;

[0022] Figure 8 This is a schematic diagram of the structure of a data reading device provided in an embodiment of the present disclosure;

[0023] Figure 9 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0024] To better understand the above-mentioned objectives, features, and advantages of this disclosure, the solutions disclosed herein will be further described below. It should be noted that, unless otherwise specified, the embodiments and features described herein can be combined with each other.

[0025] Numerous specific details are set forth in the following description in order to provide a full understanding of this disclosure, but this disclosure may also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only some, and not all, of the embodiments of this disclosure.

[0026] Data storage and / or data retrieval scenarios are involved in many fields. Taking e-commerce product detail pages as an example, it's common to call downstream services such as RPC (Remote Procedure Call Protocol) to retrieve data that changes relatively infrequently and has low consistency requirements (such as brand names and store information) for caching. This data can be cached in key-value pairs in storage systems like Redis, where the key is the key (also called the critical index) and the value is the data to be stored. Data can then be retrieved from the storage system based on the key. In many cases, the values ​​to be written to the cache are large, resulting in many large keys in storage systems like Redis. A large key is one whose corresponding value is very large (i.e., the stored data is large). Large keys pose a significant threat to service stability. For example, when retrieving data, a large value can severely impact read performance, significantly increasing service response time, especially during peak data read periods, potentially causing a substantial delay in the response time of storage systems like Redis, greatly degrading the user experience.

[0027] To address the above issues, this disclosure provides a data storage method, a data reading method, an apparatus, a device, and a medium, which are described in detail below for ease of understanding.

[0028] First, this disclosure provides a data storage method. Figure 1 This is a flowchart illustrating a data storage method provided in an embodiment of the present disclosure. The method can be executed by a data storage device, which can be implemented using software and / or hardware, and is generally integrated into an electronic device. Figure 1 As shown, the method mainly includes the following steps S102 to S108:

[0029] Step S102: In response to a storage instruction for the target data, the size of the target data is obtained. The storage instruction is used to instruct the target data to be stored in the target storage system.

[0030] The target data is the data to be stored, which can be newly added data or updated data. This disclosure does not limit the specific form and content of the target data, nor does it limit the storage method; it can be persistent storage or caching. Furthermore, the target storage system is the storage system into which the target data needs to be stored (written); this disclosure does not limit it, and it can be, for example, a Redis storage system.

[0031] Step S104: If the size of the target data is greater than the preset compression threshold, compress the target data to obtain compressed data and compression processing information.

[0032] In practical applications, a compression threshold can be set according to requirements. After obtaining the size of the target data, the size of the target data can be compared with the compression threshold. If the size of the target data is greater than the compression threshold, it indicates that the target data is large and therefore requires further compression processing. This disclosure does not limit the compression algorithm used for compression processing. The aforementioned compression processing information includes information such as the identifier of the compression algorithm used to compress the target data.

[0033] Step S106: If the size of the compressed data is greater than the preset splitting threshold, the compressed data is split to obtain multiple split data and splitting processing information.

[0034] This disclosure fully considers that even after compression, the resulting compressed data may still be large, potentially affecting subsequent reading performance and leading to longer service response times. To reliably and effectively ensure service stability, this disclosure further sets a splitting threshold. After obtaining the compressed data, its size can be compared with the splitting threshold. If the compressed data size is greater than the splitting threshold, it indicates that the compressed data is still large and therefore requires further splitting. This disclosure does not limit the splitting threshold or the splitting method. It is understood that the split data obtained after splitting will all be smaller than the splitting threshold, meaning they are all smaller data that can be directly stored.

[0035] Furthermore, this embodiment does not limit the number of data segments. In practical applications, the number of data segments mainly depends on the size of the data to be segmented (i.e., the compressed data) and a specified threshold (less than or equal to the segmentation threshold, which is not limited here), ensuring that each segmented data segment does not exceed the specified threshold until the compressed data is completely segmented. In this way, data segmentation can be flexibly implemented, and it can effectively ensure that each segmented data segment is a relatively small amount of data.

[0036] The aforementioned splitting processing information includes information such as the splitting method for splitting compressed data, and the total number of split data items.

[0037] Step S108: Based on the key index of the target data, multiple split data sets are associated and stored in the target storage system. The split data sets associated with the key index of the target data in the target storage system carry the splitting and compression processing information of the target data.

[0038] This disclosure does not limit the method of associated storage. For example, multiple split data can be stored in the same location in the storage system and the location can be identified and described. Alternatively, a unified identifier (such as the version identifier of the target data) can be attached to each split data so that when the target data is read later, all the split data obtained by splitting the target data can be directly obtained from the storage system.

[0039] To facilitate data storage and retrieval, data is typically stored in a key-value format within the storage system. That is, each piece of data (value) to be stored corresponds to a key index (key). In other words, the data to be stored is associated with its corresponding key index, allowing for direct retrieval of the associated data based on the key index. The target data is the data to be stored. In this embodiment, when the target data is large, it has been compressed and split into multiple smaller data segments stored in the target storage system. To facilitate accurate retrieval of the target data from the target storage system using its key index, these multiple data segments can be associated and stored in the target storage system based on the key index of the target data. For example, each data segment in the target storage system has its own corresponding key index, and the key index of each data segment can be constructed based on the key index of the target data. Since the target data has been split, and the target data will be retrieved from the target storage system using its key index, in practical applications, one of the split data can be selected as the split data associated with the key index of the target data. That is, the split data and the key index of the target data form a key-value pair, and the split data is used as the associated data with the key index of the target data. It carries the splitting and compression information of the target data, so that when the target data is retrieved from the target storage system based on the key index of the target data, the associated data can be found first, and then the target data can be restored based on the splitting and compression information.

[0040] In some implementations that store data in key-value format, each piece of data carries a key index (i.e., the key) and a version identifier of the target data. Since data may be updated (e.g., product details may be updated multiple times), each update corresponds to a version identifier. This embodiment of the disclosure ensures that each piece of data can be directly queried from the storage system by carrying a key index, and by carrying a version identifier of the target data, it ensures the accuracy of subsequent data retrieval. That is, only piece of data carrying the same version identifier can be combined to reconstruct the target data corresponding to that version identifier. This approach facilitates accurate and reliable data retrieval from the storage system.

[0041] Compared to related technologies that directly store data into the target storage system, the embodiments of this disclosure can set compression thresholds and splitting thresholds. When storing data, the data size is first obtained, and a hierarchical judgment is made based on the data size and the two preset thresholds. The above method can reasonably and effectively process larger data, so that the data can be stored in the target storage system in the form of smaller data, which is more conducive to subsequent data reading operations. This data storage method can effectively improve the efficiency of subsequent data reading, reduce data reading time, and help to further reduce service response time and ensure service stability.

[0042] This disclosure provides a step for associating and storing multiple split data sets in a target storage system based on a key index of the target data. For example, steps A through C can be performed as follows:

[0043] Step A: Obtain the sort order of each split data point among multiple split data points, as well as the version identifier corresponding to the target data.

[0044] In the process of splitting the target data, the target data can be split into N split data. For example, "asdfghjklqwert" can be split into 3 parts, and the first split data is "asdfgh", the second split data is "jklqwe", and the third split data is "rt". Therefore, the order of each split data in the multiple split data can be clearly known according to the split order.

[0045] Step B involves determining the key index for each split data point based on the sorting and the key indexes of the target data. For example, this can be implemented by referring to steps B1 and B2 below:

[0046] Step B1: For the first split data, use the key index of the target data as the key index of that split data. That is, the key index of the first split data is the same as the key index of the target data. This facilitates finding the first split data during subsequent data reading when searching for the target data in the target storage system such as Redis using the key index of the target data. In practical applications, the first split data can also carry information such as the algorithm identifier corresponding to the compression process and the total number of split data, so as to further find other split data based on this information and perform operations such as combining and decompressing the split data.

[0047] Step B2: For the split data that is not ranked first, construct the key index of the split data based on the key index of the target data, the sorting of the split data among the multiple split data, and the version identifier corresponding to the target data.

[0048] For example, suppose the target data is "asdfghjklqwert" and the key index of the target data is "pid_123456". After compression and splitting, it becomes three parts: "asdfgh" (sorted as 1), "jklqwe" (sorted as 2), and "rt" (sorted as 3). Suppose the version identifier (DataVersion) of the target data is 1, and the key index of the split data can be set as {key index of the target data}#{sorting of the split data among multiple split data}#{version identifier of the target data}. Then, the key index of the first split data "asdfgh" is "pid_123456", the key index of the second split data "jklqwe" is "pid_123456#2#1", and the key index of the third split data "rt" is "pid_123456#3#1".

[0049] Step C: Based on the version identifier corresponding to the target data, store each split data and its corresponding key index in the target storage system.

[0050] Each piece of data is the value, and the key index is the key. They can be stored in the target storage system in a key-value pair. Subsequent retrievals can be made directly using the key index. For example, step C above can be implemented as follows: steps C1 to C3.

[0051] Step C1: Obtain the metadata of the target data. This metadata includes: the algorithm identifier corresponding to the compression process, the total number of data segments, and the version identifier of the target data. The algorithm identifier is the identifier of the compression algorithm used; different compression algorithms have different identifiers. If the target data is not compressed, the algorithm identifier will be empty. In practical applications, fields such as compression, segmentation, and data version can be specifically set to fill in the corresponding information. If the target data is not compressed or not segmented, the corresponding fields will be empty.

[0052] Step C2 involves storing the key index corresponding to the top-ranked split data, along with the metadata of the top-ranked split data and the target data, in the target storage system.

[0053] For example, the key index corresponding to the top-ranked split data, along with the metadata of the top-ranked split data and the target data, can be stored in the target storage system according to a specified storage structure. For instance, it can be implemented with reference to the following structure:

[0054] Key (key index): pid_123456; Value (split data): asdfgh; DataVersion (version identifier): 1; TotalPart (total number of split data): 3; Compress (algorithm identifier corresponding to compression processing): 1.

[0055] The key index corresponding to the top-ranked split data is consistent with the key index of the target data. Based on this, considering that before storing the split data of the target data, the target storage system may already contain the split data of the previous version of the target data, or the split data corresponding to the target data may already exist on other devices at the same time, the following steps (1) to (3) can be used to store the top-ranked split data in the target storage system:

[0056] (1) Check in the target storage system whether there is data associated with the key index of the target data.

[0057] (2) If not found, the key index corresponding to the target data, the split data ranked first, and the metadata of the target data are associated and stored in the target storage system.

[0058] (3) If a match is found, and the version identifier of the target data is newer than the version identifier of the found data, then the found data is deleted, and the key index corresponding to the target data, the top-ranked split data, and the metadata of the target data are associated and stored in the target storage system. That is, if an older version of the target data is stored in the storage system, the current new version can replace the old version. Additionally, if a match is found, and the version identifier of the target data is older than the version identifier of the found data, then storing the target data in the target storage system is abandoned. Through the above methods, it can be fully guaranteed that the target data stored in the target storage system is the latest version. If a match is found, and the version identifier of the target data is equal to the version identifier of the found data, then there is no need to repeatedly store the top-ranked split data.

[0059] Step C3 involves associating the key index corresponding to the non-first-ranked split data with the version identifier corresponding to the non-first-ranked split data and the target data in the target storage system.

[0060] In some specific implementation examples, in order to ensure the reliability of data storage, the key index corresponding to the non-first-ranked split data and the version identifier corresponding to the non-first-ranked split data and the target data are only stored in the target storage system when specified conditions are met.

[0061] The specified conditions include either condition 1 or condition 2:

[0062] Condition 1: The key index corresponding to the target data, along with the top-ranked split data and the target data's metadata, must be successfully stored in the target storage system. This condition requires the top-ranked split data to be successfully stored in the storage system before subsequent split data not ranked first will be stored in the target storage system. This method fully considers the possibility that the top-ranked split data may not be successfully stored (e.g., if the version of the target data is older than the existing version in the target storage system, storage will be abandoned). In this case, there is no need to store the split data not ranked first.

[0063] Condition 2: Data associated with the key index of the target data is found in the target storage system, and the version identifier of the target data is the same as the version identifier of the found data. That is, if the first-ranked split data corresponding to the current version of the target data already exists in the target storage system, to ensure that the target data is stored accurately and reliably in the target storage system, subsequent non-first-ranked split data can also be stored in the target storage system. This method fully considers that although the first-ranked split data currently exists in the target storage system, other split data may not be in the target storage system due to reasons such as unsuccessful storage. Therefore, non-first-ranked split data can be further stored in the target storage system, thereby ensuring that the target storage system can completely contain all split data. Compared to additionally determining whether each other non-first-ranked split data is pre-located in the target storage system and then storing it based on the determination result, the above method provided in this disclosure embodiment is more convenient and faster.

[0064] If any of the above conditions are met, the split data that is not ranked first can be stored in the target storage system to ensure the reliability of data storage.

[0065] In practical applications, the key index corresponding to the non-first-ranked split data, the version identifier corresponding to the non-first-ranked split data and the target data can be associated and stored in the target storage system according to the specified storage structure. Taking the target data as "asdfghjklqwert" as an example, the following structure can be used as a reference:

[0066] For the second split data "jklqwe": Key (key index): pid_123456#2#1; Value (split data): jklqwe; DataVersion (version identifier): 1.

[0067] For the third split data "rt": Key (key index): pid_123456#3#1; Value (split data): rt; DataVersion (version identifier): 1.

[0068] The above is merely a simplified example and should not be considered a limitation. When storing non-first-order split data, it is associated not only with the key index and the split data, but also with a version identifier. The version identifier distinguishes the target data version to which the split data belongs, ensuring that different split data belong to the same version of the target data. Even if the target data is updated, all split data corresponding to the required version of the target data can be accurately retrieved subsequently, thus accurately restoring the required target data. Different versions of the target data correspond to different split data, but because of the version identifier, the split data corresponding to different versions of the target data will not affect each other. Furthermore, in practical applications, different versions of split data can coexist in the target storage system, or only the latest version of the split data can be retained, while older versions are deleted; no restrictions are imposed here.

[0069] Furthermore, this disclosure also provides a solution for ensuring that the size of the compressed data is not greater than a preset splitting threshold. If the size of the compressed data is not greater than the preset splitting threshold, the compressed data can be stored in the target storage system; and the compressed data carries an algorithm identifier corresponding to the compression processing (i.e., a compression algorithm identifier). That is, when the compressed data is already small, it can be stored directly. In this case, the compressed data only needs to carry a compression algorithm identifier so that when the target data is read later, the found compressed data can be directly decompressed based on the compression algorithm identifier to restore the target data.

[0070] Furthermore, this disclosure also provides a solution for situations where the size of the target data is not greater than a preset compression threshold. If the size of the target data is not greater than the preset compression threshold, the target data is directly stored in the target storage system. That is, if the target data is small, it can be stored directly.

[0071] For ease of understanding, based on the foregoing embodiments, this disclosure provides a method as follows: Figure 2 The flowchart of the data storage method shown below illustrates the process of writing target data (data that needs to be updated or added) into Redis. Please refer to steps S202 to S230 as follows:

[0072] Step S202: Determine if the length of the value to be written is greater than the compression threshold. This can be done by first obtaining the size of the target data and the compression threshold, and then comparing them. In practical applications, the target data is cached in Redis in key-value format, so the target data can be considered as the value. Therefore, determining if the length of the value to be written is greater than the compression threshold is equivalent to determining if the size of the target data is greater than the compression threshold. If it is greater, proceed to step S204; if it is less, proceed to step S210.

[0073] Step S204: Compress the value to be written. That is, compress the target data. A preset compression algorithm can be used to compress the target data, and this embodiment of the disclosure does not limit the compression algorithm.

[0074] Step S206: Determine whether the compressed value (i.e., compressed data) is greater than the splitting threshold. That is, determine whether the length of the compressed value is greater than the splitting threshold. If yes, proceed to step S208; otherwise, proceed to step S210.

[0075] Step S208: Split the compressed value. This embodiment of the disclosure does not limit the splitting method.

[0076] Step S210: Read the value corresponding to the key of the target data from Redis. The key of the target data is its critical index; this step determines whether the target data has already been stored in Redis.

[0077] As mentioned above, if the target data is large, the value read is the first part after splitting. This is because the key index of the first split data (i.e., the first part) is the same as the key index of the target data. If the target data is small, the original, unsplit value of the target data (the value of the target data or the compressed value) is read. For ease of description, this will be referred to as the value of the first part below. For unsplit target data, the value of the first part is the value corresponding to the target data, and the values ​​of other parts are considered to be non-existent.

[0078] Step S212: Determine if the data exists. That is, determine if the value corresponding to the key of the target data is currently pre-stored in Redis. If yes, proceed to step S214; otherwise, proceed to step S218.

[0079] Step S214: Determine if the version of the value to be updated is greater than the version of the value already stored in Redis. That is, determine if the version of the target data is newer than the version of the found data. If yes, proceed to step S220; otherwise, proceed to step S216.

[0080] Step S216: Determine if the version of the value to be updated is equal to the version of the value already stored in Redis. If yes, proceed to step S222. If no, it means that the version of the value to be updated is less than the version of the value already stored in Redis. In this case, abandon writing the target data, which can also be regarded as a failure to write the target data.

[0081] Step S218: If the data does not exist, the value of the first part can be directly saved to Redis. It should be noted that if the target data is large and split into multiple parts, then the first part of the split data will be written to Redis. If the target data is small and not split, then the target data or compressed data will be written, which can be considered as the target data or compressed data having only one part (also called the first part), without other parts.

[0082] In step S220, if the version of the value to be updated is greater than the version of the value already stored in Redis, the stored value can be directly updated to the value of the first part corresponding to the new version of the target data, so as to ensure that the first part of the target data stored in Redis is always the latest version.

[0083] In step S222, if the version of the value to be updated is equal to the version of the value already stored in Redis, then all parts except the first part can be directly saved in Redis, and the target data is considered to have been successfully written.

[0084] Step S224: Determine if there is a network error. If yes, the target data writing fails; otherwise, proceed to step S226.

[0085] Step S226: Determine whether the first part was saved successfully. If not, the target data writing failed; if yes, proceed to step S228.

[0086] Step S228: Save the parts other than the first part in Redis.

[0087] Step S230: Asynchronously delete the split values ​​that are not in the first part of the old version. At this point, the target data is considered to have been successfully written. This method effectively ensures that all parts of the target data stored in Redis are the latest version.

[0088] In summary, the embodiments of this disclosure can reasonably and effectively process large amounts of data, enabling the data to be stored in the target storage system in the form of smaller amounts of data. This is more conducive to subsequent data reading operations. Furthermore, when storing data, the data version is fully considered, ensuring that the target data stored in the target storage system such as Redis is the latest version.

[0089] Based on the aforementioned data storage methods, this disclosure provides a data reading method. Figure 3 This is a flowchart illustrating a data reading method provided in an embodiment of the present disclosure. The method can be executed by a data reading device, which can be implemented using software and / or hardware, and is generally integrated into an electronic device. Figure 3As shown, the method mainly includes the following steps S302 to S308:

[0090] Step S302: In response to a read instruction for the target data, locate the associated data corresponding to the key index of the target data in the target storage system. The read instruction is used to instruct the target data to be read from the target storage system.

[0091] In practical applications, target data is stored in key-value pairs in storage systems such as Redis. When it is necessary to read the target data (i.e., the value), the data related to the key index (key) can be retrieved from the target storage system (i.e., associated data).

[0092] As can be seen from the aforementioned data storage methods, when the target data is small and does not require compression or splitting, the associated data corresponding to the key index of the target data is the target data itself; when the target data is large and needs to be compressed but not split, the associated data corresponding to the key index of the target data is the compressed data; when the target data is very large and needs to be compressed and split, the associated data corresponding to the key index of the target data is the split data that is ranked first.

[0093] Step S304: If related data is found and the related data packet contains splitting and compression processing information of the target data, based on the splitting processing information and the key index of the target data, obtain multiple split data obtained by splitting the target data from the target storage system.

[0094] Multiple split data sets are stored in the target storage system as follows: In response to a storage instruction for the target data, the size of the target data is obtained; the storage instruction instructs the target data to be stored in the target storage system; if the size of the target data is greater than a preset compression threshold, the target data is compressed to obtain compressed data and compression processing information; if the size of the compressed data is greater than a preset splitting threshold, the compressed data is split to obtain multiple split data sets and splitting processing information; based on the key index of the target data, the multiple split data sets are associated and stored in the target storage system. The split data sets associated with the key index of the target data in the target storage system carry the splitting processing information and compression processing information of the target data. For details, please refer to the relevant content of the aforementioned data storage method, which will not be repeated here.

[0095] The aforementioned splitting processing information includes, for example, the total number of split data items, and the compression processing information includes, for example, the algorithm identifier corresponding to the compression processing. In practical applications, the key index corresponding to each split data item can be obtained based on the splitting processing information and the key index of the target data. Then, the corresponding split data item can be retrieved from the target storage system based on the key index corresponding to each split data item.

[0096] Step S306: Merge the obtained split data to obtain merged data.

[0097] The obtained split data can be merged according to the sorting of each split data in multiple split data sets to obtain the merged data. For example, taking the target data as "asdfghjklqwert" as an example, after compression and splitting, it gets three parts: "asdfgh" (sorted as 1), "jklqwe" (sorted as 2), and "rt" (sorted as 3). At this time, merging based on the sorting of each split data can restore "asdfghjklqwert".

[0098] Step S308: Decompress the merged data according to the compression processing information, and use the decompressed data as the target data to be read.

[0099] The above method can directly read and merge smaller, split data. Compared with directly reading larger data, it can effectively improve data reading efficiency, reduce data reading time, and help further reduce service response time and ensure service stability.

[0100] This embodiment of the disclosure takes into account that data may be updated once or multiple times. Therefore, each stored data carries a corresponding version identifier. Specifically, the associated data also includes the version identifier of the target data. Furthermore, to facilitate accurate retrieval of split data later, the splitting information contained in the associated data includes the total number of multiple split data obtained by splitting the target data. Based on this, multiple split data obtained by splitting the target data can be retrieved from the target storage system based on the splitting information and the key index of the target data. This can be achieved by referring to steps a to c below:

[0101] Step a) designates the associated data as the first split among multiple splits. As can be seen from the relevant content on data storage methods, the key index of the first split is the same as the key index of the target data. This ensures that when searching for the target data in a storage system such as Redis using the key index of the target data, the first part obtained from splitting the target data (the first split) can be found first.

[0102] Step b: Based on the key index of the target data, the total number of multiple split data, and the version identifier of the target data, construct the key index corresponding to each split data that is not ranked first.

[0103] For example, the key index for each non-first-ranked split data can be constructed as follows: {key index corresponding to the target data}#{sorting of the split data among multiple split data}#{version identifier corresponding to the target data}. For instance, if the key index corresponding to the target data is known to be "pid_123456", and the target data is split into 3 parts (i.e., the total number of split data is 3), and the version identifier corresponding to the target data is 1, then the key index corresponding to the first-ranked split data (i.e., the related data found based on the key index of the target data) is also "pid_123456", while the key index corresponding to the second-ranked split data is "pid_123456#2#1", and the key index corresponding to the third-ranked split data is "pid_123456#3#1". Using this method, the key index corresponding to each split data can be easily obtained.

[0104] Step c: Based on the key index corresponding to each non-first-ranked split data, find each non-first-ranked split data in the target storage system.

[0105] Given that the key indexes of each non-first-ranked split data are known, the corresponding split data can be found in the target storage system using these key indexes. For example, the split data corresponding to the key index "pid_123456#2#1" is "jklqwe", and the split data corresponding to the key index "pid_123456#3#1" is "rt". Additionally, the associated data found based on "pid_123456" (i.e., the first-ranked split data) is "asdfgh". With these, the version 1 split data corresponding to the target data has been accurately obtained. Merging them in order yields the target data "asdfghjklqwert".

[0106] To ensure the accuracy of data retrieval, based on the aforementioned steps, the acquired multiple split data are merged to obtain the merged data. This includes: if every split data not ranked first is found, the first-ranked split data is merged with each of the non-first-ranked split data to obtain the merged data. In other words, before merging, it is necessary to first determine whether every split data has been found. Merging only occurs when all split data has been found. This method fully considers the possibility of unsuccessful data storage (such as some split data failing to be stored successfully due to network issues). Therefore, the retrieved split data is not merged directly, but the data merging operation is only performed after ensuring that all split data has been found, thereby ensuring the accuracy of data retrieval.

[0107] In practical applications, to accurately reconstruct the target data, we can first obtain the ranking of each split data point among multiple split data points. Then, based on the ranking, we merge the multiple split data points to obtain merged data, which can then be used to further derive the target data. As mentioned earlier, the key index corresponding to the first-ranked split data point is the same as the key index of the target data. The key index corresponding to the split data points that are not ranked first is in the form of: {key index corresponding to the target data}#{ranking of this split data point among multiple split data points}#{version identifier corresponding to the target data point}. Therefore, the ranking of each split data point among multiple split data points can be determined based on the key index corresponding to that split data point. For example, if the key index "pid_123456" corresponds to a split data sorted as 1, and the found split data is "asdfgh"; if the key index "pid_123456#2#1" corresponds to a split data sorted as 2, and the found split data is "jklqwe"; and if the key index "pid_123456#3#1" corresponds to a split data sorted as 3, and the found split data is "rt", then by combining the split data in order from top to bottom, we can obtain "asdfghjklqwert". Using this method, the required data can be reconstructed efficiently and reliably based on the split data.

[0108] This disclosure also provides a solution for finding associated data where the associated data packet contains compressed information but not splitting information. Specifically, if associated data is found and the associated data packet contains compressed information but not splitting information, the associated data is decompressed according to the compressed information, and the target data is obtained based on the decompressed data. This situation indicates that the target data is slightly large and needs to be compressed during storage, but the size of the compressed data (i.e., the aforementioned compressed data) has reached the requirement for direct storage and does not need to be split. It is directly stored in the form of compressed data. Subsequently, when reading the target data, the associated data (i.e., the compressed data) is first found in the target storage system, and then decompressed, which is convenient and fast.

[0109] This disclosure also provides a solution for finding related data that contains neither compression nor splitting information. Specifically, if related data is found and does not contain either compression or splitting information, the related data is directly used as the target data to be read. This situation indicates that the target data is small in size and can be stored directly without further compression or splitting. Subsequently, when reading the target data, the related data found in the target storage system is the target data.

[0110] Furthermore, this disclosure fully considers the possibility that the target data may not have been previously stored in the target storage system, or that not all the corresponding split data may have been successfully stored in the target storage system. Therefore, it also provides solutions for not finding related data or not obtaining multiple split data corresponding to the target data. Specifically, if related data is not found or multiple split data corresponding to the target data are not obtained, the target data is directly read from the original data source. The original data source is the original location where the target data was obtained, such as calling the downstream service again to query the target data. This method can effectively ensure that the required target data can be successfully read in the end. In practical applications, the read target data can also be further stored in the target storage system so that the target data can be directly obtained from the target storage system in the future.

[0111] To facilitate understanding, based on the foregoing embodiments, this disclosure provides a specific implementation example for reading target data. In this implementation example, the aforementioned data storage method can be used to store the target data in a cache database such as Redis. Further details can be found in... Figure 4 The flowchart of the data reading method shown mainly includes the following steps S402 to S426:

[0112] Step S402: Query the local cache. That is, when reading target data, you can first try to query the local cache.

[0113] Step S404: Determine if the target data was found. If yes, confirm that the target data was successfully read; otherwise, proceed to step S406.

[0114] Step S406: Query Redis. That is, if the target data does not exist in the local cache, the target data can be queried through the storage system Redis (also known as the cache middleware).

[0115] Step S408: Determine if related data was found. In practical applications, Redis stores data in key-value format. The target data (value) can be found using the key index corresponding to the target data. The data corresponding to the key index is the related data. If yes, proceed to step S410; otherwise, proceed to step S422.

[0116] Step S410: Determine whether the data has been split. Specifically, this can be determined based on the associated data. In practical applications, if the associated data packet contains splitting information, it is confirmed that the target data has been split. If so, proceed to step S412; otherwise, proceed to step S418.

[0117] Step S412: Query the other parts of the split in Redis. That is, if it is confirmed that the target data has been split, the retrieved related data is the first split data (i.e., the first part obtained from the split). Then, it is necessary to further query the other parts of the split. Specifically, based on the key index corresponding to the target data, the split information, and the version identifier of the target data, key indexes corresponding to each split part can be constructed. Then, each split part can be queried based on its corresponding key index.

[0118] Step S414: Determine if all split parts have been found. If yes, proceed to step S416; otherwise, proceed to step S422.

[0119] Step S416: Merge all the split parts (i.e., the multiple split data corresponding to the aforementioned target data), and then execute step S420. It is understandable that, as described in the aforementioned data storage method, the target data needs to be compressed before splitting; that is, it is compressed before splitting. Therefore, the merged data is essentially compressed data and needs to be decompressed.

[0120] Step S418: If the target data has not been split, determine whether the target data has been compressed. Specifically, this can be determined based on the associated data. In practical applications, if the associated data packet contains compression information (such as the algorithm identifier corresponding to the compression process), it is confirmed that the target data has been compressed. If so, proceed to step S420; otherwise, proceed to step S426.

[0121] Step S420, decompression. That is, the compressed data can be decompressed based on the compression information, and then step S426 is executed.

[0122] Step S422: If no related data is found in Redis or no split parts are found in Redis, then downstream services are called to query, that is, the target data is queried through the original data source.

[0123] Step S424: Write to Redis. Specifically, the target data retrieved from the original data source is written to Redis so that it can be directly retrieved from Redis later. Then, step S426 is executed.

[0124] Step S426: Write to local cache. Specifically, the retrieved target data is written to the local cache.

[0125] By using the above method, data in the local cache is queried first. If the target data is not found in the local cache, it is then queried through Redis. Furthermore, it can be determined whether the related data retrieved from Redis has been split / compressed, and appropriate processing methods (finding and merging split data, decompression, etc.) are used to restore the data. This ensures that the required data is read accurately and reliably. Since the data stored in Redis is compressed or split and is usually smaller, it can effectively improve data reading efficiency and reduce data reading time compared to directly reading larger original data. This helps to further reduce service response time and ensure service stability.

[0126] To facilitate understanding of the aforementioned data storage method and data retrieval method, embodiments of this disclosure provide, as follows: Figure 5 The diagram illustrates a successful data processing step, showcasing three key stages: data splitting, data storage, and data retrieval. During the data splitting stage, the target data (original data) can be updated multiple times. Figure 5 The example shows three updates, meaning the original data can be updated sequentially to three versions: V1, V2, and V3. Assuming the target data is large, each version needs to be split into three parts: part1, part2, and part3. It should be noted that... Figure 5This is just an example. In practical applications, the amount of data split across different versions can be the same or different. For example, the first update version might be split into 3 parts, and the second update version into 4 parts. Each split is identified based on the version of the target data. For instance, the 3 parts in the first update version might all be v1, and the 3 parts in the second update version might all be v2. This version identifier ensures that different split parts belong to the same original data. In other words, the version identifier effectively isolates different versions corresponding to the same key (key or critical index), preventing split data from affecting each other. Furthermore, multiple parts split from the same original data share the same version identifier, thus ensuring data consistency and reliability.

[0127] It should be noted that raw data, such as target data, can be split into multiple parts based on a preset threshold. Each part exists independently at the data level, and the first part (part1) has the same key as the target data. This ensures that when querying data based on the target data key, the corresponding part1 can be found first. In addition, part1 also contains metadata (version information, splitting information, compression information, etc. of the target data) to facilitate further searching of other parts based on the metadata. The splitting logic can be retried each time the target data changes. In practical applications, only the latest version of the split part can be retained. For example, after the third update, the part1 after the third update can directly replace the part1 after the second update. In this case, only part1-v3 is associated with the target data key in Redis.

[0128] In the data storage phase, theoretically, the latest updated version is stored, such as the third updated version V3. During data retrieval, part1-v3 can be read first based on the key of the target data. Then, based on the metadata corresponding to part1-v3, the keys corresponding to part2-v3 and part3-v3 are constructed. Subsequently, based on the keys corresponding to part2-v3 and part3-v3, part2-v3 and part3-v3 can be read from Redis. Finally, merging part1-v3, part2-v3, and part3-v3 yields the correct result. Figure 5 This is a simple illustration of successful data storage and successful data retrieval.

[0129] However, considering the possibility of partial storage failures during data storage, there could be various reasons for this. These could include network or storage system issues when storing a particular partition, or problems arising when replacing an existing partition with the latest version, resulting in the incorrect storage of the newer version's data. (See also...) Figure 6The diagram illustrating a data processing failure shows a scenario where data storage fails. For example, when storing the third updated version, although part1-v2 is replaced with part1-v3, part2-v3 and part3-v3 fail to be stored in Redis. Redis still stores part2-v2 and part3-v2. This causes subsequent data retrieval steps to only read part1-v3, failing to retrieve part2-v3 and part3-v3, resulting in data retrieval failure. In practical applications, it's necessary to determine if all split parts corresponding to the same version have been retrieved. If so, a merge operation is performed; otherwise, a process of re-retrieval is triggered, such as re-calling downstream services to obtain data from the original data source.

[0130] like Figure 5 As shown, the above method effectively reduces data retrieval time by splitting data, thereby ensuring service stability, and fully taking into account the existence of issues such as... Figure 6 The example shown illustrates a scenario where data is split but storage fails. However, after a data read failure, further verification can be performed based on the data read results. This allows for the re-acquisition of data if it is confirmed that not all split data of the same version has been obtained, thus effectively ensuring the reliability of data reads.

[0131] Corresponding to the aforementioned data storage method, this disclosure provides a data storage device. Figure 7 This is a schematic diagram of a data storage device provided in an embodiment of the present disclosure. The device can be implemented by software and / or hardware, and is generally integrated into an electronic device, such as... Figure 7 As shown, it includes:

[0132] Size acquisition module 702 is used to acquire the size of target data in response to a storage instruction for target data; the storage instruction is used to instruct the target data to be stored in a target storage system.

[0133] The compression processing module 704 is used to compress the target data if the size of the target data is greater than a preset compression threshold, so as to obtain compressed data and compression processing information.

[0134] The splitting processing module 706 is used to split the compressed data if the size of the compressed data is greater than a preset splitting threshold, so as to obtain multiple split data and splitting processing information.

[0135] The associated storage module 708 is used to associate and store multiple split data in the target storage system based on the key index of the target data; wherein, the split data associated with the key index of the target data in the target storage system carries the splitting processing information and compression processing information of the target data.

[0136] The aforementioned device can process large amounts of data in a reasonable and effective manner, enabling the data to be stored in the storage system in the form of smaller data, which is more conducive to subsequent data reading operations. This data storage method can effectively improve the efficiency of subsequent data reading, reduce data reading time, and help to further reduce service response time and ensure service stability.

[0137] In some implementations, the associated storage module 708 is specifically used to: obtain the order of each of the split data in the plurality of split data, and the version identifier corresponding to the target data; determine the key index of each of the split data based on the order and the key index of the target data; and associate and store each of the split data and the corresponding key index in the target storage system based on the version identifier corresponding to the target data.

[0138] In some implementations, the associated storage module 708 is specifically used to: for the first-ranked split data, use the key index of the target data as the key index of the split data; for the split data that is not ranked first, construct the key index of the split data based on the key index of the target data, the order of the split data among the multiple split data, and the version identifier corresponding to the target data.

[0139] In some implementations, the associated storage module 708 is specifically used to: acquire metadata of the target data; wherein the metadata includes: the algorithm identifier corresponding to the compression process, the total number of the multiple split data, and the version identifier corresponding to the target data; associate and store the key index corresponding to the first-ranked split data, the metadata of the first-ranked split data and the target data in the target storage system; associate and store the key index corresponding to the non-first-ranked split data, the version identifier of the non-first-ranked split data and the target data in the target storage system.

[0140] In some implementations, the key index corresponding to the top-ranked split data is consistent with the key index of the target data; the associated storage module 708 is specifically used to: search in the target storage system for whether there is currently any data associated with the key index of the target data; if not found, associate and store the key index corresponding to the target data, the top-ranked split data, and the metadata of the target data in the target storage system; if found, and the version identifier corresponding to the target data is newer than the version identifier corresponding to the found data, delete the found data, and associate and store the key index corresponding to the target data, the top-ranked split data, and the metadata of the target data in the target storage system.

[0141] In some implementations, the associated storage module 708 is specifically used to: if a specified condition is met, associate and store the key index corresponding to the non-first-ranked split data, the version identifier corresponding to the non-first-ranked split data, and the target data in the target storage system; wherein, the specified condition includes: successfully storing the key index corresponding to the target data, the metadata of the first-ranked split data, and the target data in the target storage system; or, finding data in the target storage system that is currently associated with the key index of the target data, and the version identifier corresponding to the target data is the same as the version identifier corresponding to the found data.

[0142] In some embodiments, the apparatus further includes a discard storage module for discarding the storage of the target data in the target storage system if the version identifier corresponding to the target data is older than the version identifier corresponding to the found data.

[0143] In some embodiments, the apparatus further includes a compression storage module, used to store the compressed data in the target storage system if the size of the compressed data is not greater than the preset splitting threshold; and the compressed data carries an algorithm identifier corresponding to the compression process.

[0144] The data storage device provided in this disclosure can execute the data storage method provided in any embodiment of this disclosure, and has the corresponding functional modules and beneficial effects of executing the method.

[0145] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the above-described device embodiments can be referred to the corresponding process in the method embodiments, and will not be repeated here.

[0146] Corresponding to the aforementioned data reading method, this disclosure provides a data reading device. Figure 8 This is a schematic diagram of a data reading device provided in an embodiment of the present disclosure. The device can be implemented by software and / or hardware, and is generally integrated into an electronic device, such as... Figure 8 As shown, it includes:

[0147] The associated data lookup module 802 is used to, in response to a read instruction for target data, look up associated data corresponding to the key index of the target data in the target storage system; the read instruction is used to instruct the target data to be read from the target storage system.

[0148] The split data acquisition module 804 is used to acquire multiple split data obtained by splitting the target data from the target storage system based on the split processing information and the key index of the target data if related data is found and the related data packet contains split processing information and compression processing information of the target data.

[0149] The merging processing module 806 is used to merge multiple split data sets to obtain merged data.

[0150] The decompression processing module 808 is used to decompress the merged data according to the compression processing information and use the decompressed data as the target data to be read.

[0151] The aforementioned device can directly read and merge smaller, fragmented data. Compared to directly reading larger data, it can effectively improve data reading efficiency, reduce data reading time, and help further reduce service response time and ensure service stability.

[0152] In some implementations, the associated data further includes a version identifier of the target data; the splitting processing information includes the total number of multiple split data obtained by splitting the target data; the split data acquisition module 804 is specifically used to: regard the associated data as the first split data among the multiple split data; construct a key index corresponding to each non-first split data based on the key index of the target data, the total number of the multiple split data and the version identifier of the target data; and search for each non-first split data in the target storage system based on the key index corresponding to each non-first split data.

[0153] In some implementations, the merging processing module 806 is specifically used to: if each split data that is not ranked first is found, merge the split data that is ranked first with each split data that is not ranked first to obtain merged data.

[0154] In some implementations, the merging processing module 806 is specifically used to: obtain the order of each of the split data in the plurality of split data; and merge the plurality of split data based on the order to obtain merged data.

[0155] In some embodiments, the above-described apparatus further includes a data source acquisition module, used to read the target data from the original data source if the associated data is not found or multiple split data corresponding to the target data are not acquired.

[0156] In some embodiments, the above-described apparatus further includes a decompression and reading module, used to decompress the associated data according to the compression information if the associated data is found and the associated data packet contains the compression information but does not contain the splitting information, and to obtain the read target data based on the decompressed data.

[0157] In some implementations, the plurality of split data are stored in the target storage system in the following manner: in response to a storage instruction for the target data, the size of the target data is obtained; the storage instruction is used to instruct the target data to be stored in the target storage system; if the size of the target data is greater than a preset compression threshold, the target data is compressed to obtain compressed data and compression processing information; if the size of the compressed data is greater than a preset splitting threshold, the compressed data is split to obtain splitting processing information to obtain the plurality of split data; based on the key index of the target data, the plurality of split data are associated and stored in the target storage system; wherein, the split data associated with the key index of the target data in the target storage system carries the splitting processing information and compression processing information of the target data.

[0158] The data reading device provided in this disclosure can execute the data reading method provided in any embodiment of this disclosure, and has the corresponding functional modules and beneficial effects of executing the method.

[0159] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the above-described device embodiments can be referred to the corresponding process in the method embodiments, and will not be repeated here.

[0160] This disclosure provides an electronic device, which includes: a processor; a memory for storing processor-executable instructions; and a processor for reading executable instructions from the memory and executing the instructions to implement the data storage method or the data reading method described above.

[0161] Figure 9 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. Figure 9 As shown, the electronic device 900 includes one or more processors 901 and memory 902.

[0162] The processor 901 may be a central processing unit (CPU) or other form of processing unit with data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 900 to perform desired functions.

[0163] The memory 902 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 901 may execute the program instructions to implement the data storage method, data reading method, and / or other desired functions described in the embodiments of this disclosure above. Various contents such as input signals, signal components, and noise components may also be stored in the computer-readable storage medium.

[0164] In one example, the electronic device 900 may also include an input device 903 and an output device 904, which are interconnected via a bus system and / or other forms of connection mechanism (not shown).

[0165] In addition, the input device 903 may also include, for example, a keyboard, a mouse, etc.

[0166] The output device 904 can output various information to the outside, including determined distance information, direction information, etc. The output device 904 may include, for example, a display, a speaker, a printer, and a communication network and its connected remote output devices, etc.

[0167] Of course, for the sake of simplicity, Figure 9 Only some of the components of the electronic device 900 relevant to this disclosure are shown, omitting components such as buses, input / output interfaces, etc. In addition, the electronic device 900 may include any other suitable components depending on the specific application.

[0168] In addition to the methods and devices described above, embodiments of this disclosure may also be computer program products, which include computer program instructions that, when executed by a processor, cause the processor to perform the data storage method or data retrieval method provided in the embodiments of this disclosure.

[0169] The computer program product can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of this disclosure. The programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on a user's computing device, partially on a user's computing device, as a standalone software package, partially on a user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0170] Furthermore, embodiments of this disclosure may also be computer-readable storage media storing computer program instructions thereon, which, when executed by a processor, cause the processor to perform the data storage method or data retrieval method provided in the embodiments of this disclosure.

[0171] The computer-readable storage medium may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may, for example, include, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0172] This disclosure also provides a computer program product, including a computer program / instructions, which, when executed by a processor, implement the data storage method or data reading method in this disclosure.

[0173] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0174] The above description is merely a specific embodiment of this disclosure, enabling those skilled in the art to understand or implement it. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this disclosure. Therefore, this disclosure is not to be limited to the embodiments described herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A data storage method, characterized in that, include: In response to a storage instruction for target data, the size of the target data is obtained; the storage instruction is used to instruct the target data to be stored in a target storage system. If the size of the target data is greater than a preset compression threshold, the target data is compressed to obtain compressed data and compression processing information. If the size of the compressed data is greater than a preset splitting threshold, the compressed data is split to obtain multiple split data and splitting processing information; Based on the key index of the target data, multiple split data are associated and stored in the target storage system; wherein, the split data associated with the key index of the target data in the target storage system carries the splitting processing information and compression processing information of the target data; wherein, the target storage system associates and stores: the key index corresponding to the first split data among the multiple split data, the metadata of the first split data and the target data, the metadata including: the algorithm identifier corresponding to the compression processing, the total number of the multiple split data, and the version identifier corresponding to the target data; the target storage system also associates and stores: the key index corresponding to the non-first split data among the multiple split data, the version identifier of the non-first split data and the target data.

2. The method according to claim 1, characterized in that, The step of associating and storing multiple split data sets in the target storage system based on the key index of the target data includes: Obtain the sort order of each of the split data in the plurality of split data, and the version identifier corresponding to the target data; Based on the sorting and the key indexes of the target data, determine the key indexes for each of the split data; Based on the version identifier corresponding to the target data, each of the split data and its corresponding key index are associated and stored in the target storage system.

3. The method according to claim 2, characterized in that, The step of determining the key index of each of the split data based on the sorting and the key index of the target data includes: For the top-ranked split data, the key index of the target data is used as the key index of that split data. For split data that is not ranked first, a key index for the split data is constructed based on the key index of the target data, the order of the split data among the multiple split data, and the version identifier corresponding to the target data.

4. The method according to claim 2, characterized in that, The step of associating and storing each split data and its corresponding key index in the target storage system based on the version identifier corresponding to the target data includes: Obtain the metadata of the target data; The key index corresponding to the top-ranked split data, along with the metadata of the top-ranked split data and the target data, are stored in the target storage system. The key index corresponding to the non-first-ranked split data, the version identifier corresponding to the non-first-ranked split data and the target data are associated and stored in the target storage system.

5. The method according to claim 4, characterized in that, The key index corresponding to the top-ranked split data is consistent with the key index of the target data; The step of associating and storing the key index corresponding to the top-ranked split data, the metadata of the top-ranked split data, and the target data in the target storage system includes: Search the target storage system to see if there is currently any data associated with the key index of the target data; If not found, the key index corresponding to the target data, the top-ranked split data, and the metadata of the target data are associated and stored in the target storage system. If a match is found, and the version identifier corresponding to the target data is newer than the version identifier corresponding to the found data, then the found data is deleted, and the key index corresponding to the target data, the split data ranked first, and the metadata of the target data are associated and stored in the target storage system.

6. The method according to claim 5, characterized in that, The step of associating and storing the key index corresponding to the non-first-ranked split data, the version identifier corresponding to the non-first-ranked split data, and the target data in the target storage system includes: If the specified conditions are met, the key index corresponding to the non-first-ranked split data, the version identifier corresponding to the non-first-ranked split data and the target data will be associated and stored in the target storage system. The specified conditions include: successfully storing the key index corresponding to the target data, the top-ranked split data, and the metadata of the target data in the target storage system; or, finding data in the target storage system that is currently associated with the key index of the target data, and the version identifier corresponding to the target data is the same as the version identifier corresponding to the found data.

7. The method according to claim 5, characterized in that, The method further includes: If the target data is found, and the version identifier corresponding to the target data is older than the version identifier corresponding to the found data, then the target data is abandoned from being stored in the target storage system.

8. The method according to claim 1, characterized in that, The method further includes: If the size of the compressed data is not greater than the preset splitting threshold, the compressed data is stored in the target storage system; and the compressed data carries the algorithm identifier corresponding to the compression process.

9. A data reading method, characterized in that, include: In response to a read command for target data, locate the associated data corresponding to the key index of the target data in the target storage system; The read instruction is used to instruct the reading of the target data from the target storage system; If the associated data is found and the associated data packet contains the splitting and compression processing information of the target data, based on the splitting processing information and the key index of the target data, multiple split data obtained by splitting the target data are retrieved from the target storage system; wherein, the target storage system stores in association: the key index corresponding to the first split data among the multiple split data, the metadata of the first split data and the target data, the metadata including: the algorithm identifier corresponding to the compression processing, the total number of the multiple split data, and the version identifier corresponding to the target data; the target storage system also stores in association: the key index corresponding to the non-first split data among the multiple split data, the version identifier of the non-first split data and the target data. The obtained multiple split data are merged to obtain merged data; The merged data is decompressed according to the compression information, and the decompressed data is used as the target data to be read.

10. The method according to claim 9, characterized in that, The associated data also includes the version identifier of the target data; the splitting processing information includes the total number of multiple split data obtained by splitting the target data; Based on the splitting processing information and the key index of the target data, the steps of obtaining multiple split data obtained by splitting the target data from the target storage system include: The associated data is selected as the first split data among the multiple split data; Based on the key index of the target data, the total number of the multiple split data, and the version identifier of the target data, construct the key index corresponding to each split data that is not ranked first; Based on the key index corresponding to each non-first-ranked split data, find each non-first-ranked split data in the target storage system.

11. The method according to claim 10, characterized in that, The step of merging the acquired multiple split data sets to obtain merged data includes: If every split data point that is not ranked first is found, the split data that is ranked first is merged with every split data point that is not ranked first to obtain the merged data.

12. The method according to claim 9, characterized in that, The steps of merging the acquired multiple split data to obtain merged data include: Obtain the sort order of each of the split data points among the plurality of split data points; Based on the sorting, the multiple split data are merged to obtain merged data.

13. The method according to claim 9, characterized in that, The method further includes: If the associated data is not found or multiple split data corresponding to the target data are not obtained, the target data is read from the original data source.

14. The method according to claim 9, characterized in that, The method further includes: If the associated data is found and the associated data packet contains the compressed information but not the splitting information, the associated data is decompressed according to the compressed information, and the target data is obtained based on the decompressed data.

15. The method according to any one of claims 9 to 14, characterized in that, The multiple split data are stored in the target storage system in the following manner: In response to a storage instruction for the target data, the size of the target data is obtained; the storage instruction is used to instruct the target data to be stored in a target storage system. If the size of the target data is greater than a preset compression threshold, the target data is compressed to obtain compressed data and compression processing information. If the size of the compressed data is greater than a preset splitting threshold, the compressed data is split to obtain the multiple split data and splitting processing information; Based on the key index of the target data, the multiple split data are associated and stored in the target storage system; wherein, the split data associated with the key index of the target data in the target storage system carries the splitting processing information and compression processing information of the target data.

16. A data storage device, characterized in that, include: A size acquisition module is used to acquire the size of the target data in response to a storage instruction for the target data; the storage instruction is used to instruct the target data to be stored in a target storage system; The compression processing module is used to compress the target data if the size of the target data is greater than a preset compression threshold, so as to obtain compressed data and compression processing information. The splitting processing module is used to split the compressed data if the size of the compressed data is greater than a preset splitting threshold, so as to obtain multiple split data and splitting processing information. An associated storage module is used to associate and store multiple split data sets in the target storage system based on a key index of the target data. The split data sets associated with the key index of the target data in the target storage system carry splitting and compression processing information of the target data. The target storage system associates and stores: the key index corresponding to the first split data set among the multiple split data sets, the metadata of the first split data set and the target data, the metadata including: the algorithm identifier corresponding to the compression processing, the total number of the multiple split data sets, and the version identifier corresponding to the target data. The target storage system also associates and stores: the key index corresponding to a non-first split data set among the multiple split data sets, the version identifier of the non-first split data set and the target data.

17. A data reading device, characterized in that, include: The associated data lookup module is used to, in response to a read command for target data, search for associated data corresponding to the key index of the target data in the target storage system; the read command is used to instruct the target data to be read from the target storage system. The split data acquisition module is used to, if the associated data is found and the associated data packet contains split processing information and compression processing information of the target data, obtain multiple split data obtained by splitting the target data from the target storage system based on the split processing information and the key index of the target data; wherein, the target storage system stores in association: the key index corresponding to the first split data among the multiple split data, the metadata of the first split data and the target data, the metadata including: the algorithm identifier corresponding to the compression processing, the total number of the multiple split data, and the version identifier corresponding to the target data; the target storage system also stores in association: the key index corresponding to the non-first split data among the multiple split data, the version identifier of the non-first split data and the target data. The merging processing module is used to merge the acquired multiple split data to obtain merged data; The decompression processing module is used to decompress the merged data according to the compression processing information, and use the decompressed data as the target data read.

18. An electronic device, characterized in that, The electronic device includes: processor; Memory used to store the processor's executable instructions; The processor is configured to read the executable instructions from the memory and execute the instructions to implement the data storage method according to any one of claims 1-8 or the data reading method according to any one of claims 9-15.

19. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which is used to execute the data storage method according to any one of claims 1-8 or the data reading method according to any one of claims 9-15.

Images