A data storage method, a data reading method and a device

Abstract

The present disclosure provides a data storage method, a data reading method and device, the method comprising: based on the data volume fragmentation threshold, the original data is fragmented, and a plurality of fragmented data of the original data is obtained, the data volume fragmentation threshold is less than the preset data capacity; based on the storage address of the plurality of fragmented data and the combination relationship of the plurality of fragmented data, the order identification of the original data is determined; the order identification and the plurality of fragmented data are stored in different machines. The method provided by the present disclosure can reduce the situation of insufficient machine cache and network congestion when storing the plurality of fragmented data in different machines, thereby improving the data storage efficiency, without the need to increase high-performance hardware devices and manual intervention, thereby reducing the cost, and when storing the plurality of fragmented data, the online system does not need to be reconstructed, only the data is processed, the online risk is reduced, and the business loss caused by online failure is reduced.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to a data storage method, a data reading method, and an apparatus. Background Technology

[0002] Currently, with the rapid development of the Internet, the amount of data is growing rapidly, which puts excessive pressure on server caching components, increases network overhead, and leads to insufficient caching interfaces, or even service avalanche, making it impossible to provide normal services to users.

[0003] Among the relevant technologies, two solutions are selected: one is to improve the hardware by adding caching components or increasing network bandwidth to enhance caching capabilities; the other is to improve the system by significantly reconstructing the existing system and splitting the business processes with excessively large data volumes. Summary of the Invention

[0004] According to one aspect of this disclosure, a data storage method is provided, the method comprising:

[0005] The original data is divided into multiple fragments based on a data volume fragmentation threshold, wherein the data volume fragmentation threshold is less than a preset data capacity.

[0006] Based on the identifiers of multiple data fragments and the combination relationships of the multiple data fragments, an ordering identifier is determined;

[0007] The ordering identifier and the multiple fragmented data are stored on different machines.

[0008] According to another aspect of this disclosure, a data reading method is provided, the method comprising:

[0009] In response to a data retrieval request, an ordering identifier matching the data retrieval request is obtained from the distributed storage machine based on the data retrieval request. The ordering identifier is used to indicate the combination relationship of different data shards. The ordering identifier and the different data shards are stored in different machines, and the data volume of each data shard is less than a preset data capacity.

[0010] Based on the ordered identifier, the fragmented data stored in different machines are combined to obtain the data reading result.

[0011] According to another aspect of this disclosure, a data storage device is provided, the device comprising:

[0012] The sharding module is used to shard the original data based on a data volume sharding threshold to obtain multiple shards of the original data, wherein the data volume sharding threshold is less than a preset data capacity.

[0013] The determination module is used to determine the ordering identifier based on the identifiers of multiple data fragments and the combination relationship of the multiple data fragments;

[0014] A storage module is used to store the ordering identifier and multiple fragmented data in different machines.

[0015] According to another aspect of this disclosure, a data reading apparatus is provided, the apparatus comprising:

[0016] A response module is used to respond to a data retrieval request by obtaining an ordering identifier matching the data retrieval request from a distributed storage machine. The ordering identifier is used to indicate the combination relationship of different data shards. The ordering identifier and the different data shards are stored in different machines, and the data volume of each data shard is less than a preset data capacity.

[0017] The acquisition module is used to combine the fragmented data stored in different machines based on the ordered identifier to obtain the data reading result.

[0018] According to another aspect of this disclosure, an electronic device is provided, comprising:

[0019] Processor; and,

[0020] Memory for stored programs;

[0021] The program includes instructions that, when executed by the processor, cause the processor to perform the method according to an exemplary embodiment of the present disclosure.

[0022] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is provided, wherein the non-transitory computer...

[0023] A readable storage medium stores computer instructions for causing the computer to perform the method according to exemplary embodiments of the present disclosure.

[0024] One or more technical solutions provided in the exemplary embodiments of this disclosure can shard the original data based on a data volume sharding threshold, thereby obtaining multiple shards of the original data, where each data volume sharding threshold is less than a preset data capacity. Based on this, when storing data, for original data exceeding the preset data capacity, the original data can be split according to the data volume scale to obtain multiple shards. Therefore, as the business volume increases, the data volume of the shards will not change. In this case, storing multiple shards on different machines reduces the risk of insufficient machine caching and network congestion, thereby improving data storage efficiency. Furthermore, it eliminates the need for additional high-performance hardware and manual intervention, thus reducing costs.

[0025] Furthermore, the exemplary embodiments of this disclosure can also determine the ordering identifier of the original data based on the storage addresses of multiple data shards and the combination relationship of the multiple data shards, and store the ordering identifier in different machines. Based on this, in response to a data retrieval request, the ordering identifier matching the data retrieval request can be obtained from the distributed storage machine, and the data shards stored in different machines can be combined based on the ordering identifier to obtain the final data retrieval result. It is evident that the exemplary embodiments of this disclosure, when storing multiple data shards, do not require reconstruction of the online system; only data processing is performed, thereby reducing deployment risks and minimizing business losses caused by online failures. Attached Figure Description

[0026] The accompanying drawings, which are included to provide a further understanding of this disclosure and form part of this disclosure, illustrate exemplary embodiments of the present disclosure and are used to explain the disclosure, but do not constitute an undue limitation of the disclosure. In the drawings:

[0027] Figure 1 A basic flowchart of a data storage method according to an exemplary embodiment of the present disclosure is shown;

[0028] Figure 2 A flowchart illustrating a data fragmentation method according to an exemplary embodiment of the present disclosure is shown;

[0029] Figure 3 A schematic diagram illustrating a self-describing structure according to exemplary embodiments of the present disclosure is shown;

[0030] Figure 4 A basic flowchart of a data reading method according to an exemplary embodiment of the present disclosure is shown;

[0031] Figure 5 A schematic block diagram of the functional modules of a data storage device according to an exemplary embodiment of the present disclosure is shown;

[0032] Figure 6 A schematic block diagram of the functional modules of a data reading apparatus according to an exemplary embodiment of the present disclosure is shown;

[0033] Figure 7 A schematic block diagram of a chip according to an exemplary embodiment of the present disclosure is shown;

[0034] Figure 8 A structural block diagram of an exemplary electronic device that can be used to implement embodiments of the present disclosure is shown. Detailed Implementation

[0035] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0036] It should be understood that the steps described in the method embodiments of this disclosure may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of this disclosure is not limited in this respect.

[0037] The term "comprising" and its variations as used herein are open-ended, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the description below. It should be noted that the concepts of "first", "second", etc., used in this disclosure are only used to distinguish different devices, modules, or units, and are not intended to limit the order of functions performed by these devices, modules, or units or their interdependencies.

[0038] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0039] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0040] Before introducing the embodiments of this disclosure, the relevant terms involved in the embodiments of this disclosure are first defined as follows:

[0041] Remote Dictionary Server (Redis): Redis is a high-performance key-value database. The key is figuratively called a secret key. Redis provides numerous commands to manipulate this "key," thereby managing the stored data. The key can be seen as a variable representing the value; by searching the key, the corresponding value can be found. The emergence of Redis largely compensates for the shortcomings of distributed high-speed caching systems and other key-value stores, and in some cases, it can serve as a good supplement to relational databases. It provides clients for Java, C / C++, C#, PHP, JavaScript, Perl, Objective-C, Python, Ruby, Erlang, and other languages, making it very convenient to use.

[0042] In Redis, a Bigkey refers to a key whose value is very large. Operating on Bigkeys is typically time-consuming, increasing the likelihood of Redis blocking. Therefore, it's crucial to identify Bigkeys early to avoid impacting production environments. Redis clients have built-in Bigkey lookup functionality, which iterates through all keys on the server. This process consumes significant server resources, impacting production performance. Furthermore, the client's built-in Bigkey lookup only retrieves the largest key of each type, failing to identify more Bigkeys and thus not meeting the needs of production environments. High concurrency refers to a situation where a system experiences a large number of requests within a short period, primarily occurring in web systems with a concentrated surge in access and requests (e.g., the 12306 train ticket rush; the Tmall Double Eleven shopping festival). This situation causes the system to perform numerous operations during this period, such as resource requests and database operations.

[0043] Caching technology generally refers to using a faster storage device to store frequently used data for quick access by users. Users do not need to interact with the slower storage device every time, thus improving access efficiency.

[0044] Distributed caching, a feature that improves the speed of instruction and data retrieval, has been widely used in the field of distributed computing as local computer systems have expanded into distributed systems. It refers to storing frequently accessed data in a distributed environment or system close to users and applications, preferably on faster devices, to reduce latency in remote data transmission and allow users and applications to quickly access the data they need.

[0045] Network overhead refers to the proportion of redundant data in the source data during network transmission, which is necessary for signal transmission and requires data format transformation.

[0046] The avalanche effect, or cascading failure of services, refers to the phenomenon where service calls between microservices cascade due to the failure of one service. It describes the process where the unavailability of a provider leads to the unavailability of the consumer, and this unavailability gradually amplifies.

[0047] A single point of failure is a common problem in distributed systems that typically use a master-slave architecture, where a host connects to multiple processing nodes. The master node is responsible for distributing tasks, while the slave nodes are responsible for processing business logic. When the master node fails, the entire system will fail. This type of failure is called a single point of failure.

[0048] The lifecycle of data refers to the process from the generation or acquisition of a collection of data to its destruction.

[0049] In recent years, in scenarios involving high-performance, high-concurrency internet servers, the volume of individual business operations has grown rapidly. Consequently, the amount of data carried by the server has also increased dramatically, as has the amount of data that each storage space needs to cache. This increase in data volume puts greater pressure on caching components and increases network overhead, which in turn leads to a sharp decline in interface service capabilities, causing a service avalanche effect and making it impossible to provide services normally.

[0050] In related technologies, the problem of excessive data caching has been addressed from a hardware or system perspective. However, from a hardware perspective, improving hardware performance or adding caching components to increase network bandwidth presents challenges. High-performance hardware is prohibitively expensive, network bandwidth is limited, and single points of failure are common. From a system perspective, significant system refactoring can enable the modified system to handle larger data volumes. However, such substantial modifications easily introduce deployment risks, leading to system failures and business losses.

[0051] To address the aforementioned issues, this exemplary embodiment provides a data storage method and a data retrieval method. These methods can segment excessively large amounts of stored data, assign ordered identifiers to multiple data segments, and store these segments on different machines to increase data caching capabilities. Furthermore, in response to a data retrieval request, the multiple data segments are combined according to their ordered identifiers, enabling fast and convenient data storage and retrieval. Moreover, the hardware can be reused without incurring additional hardware costs, and the original cache can be fault-free without human intervention.

[0052] Figure 1 A basic flowchart of a data storage method according to an exemplary embodiment of the present disclosure is shown. Figure 1 As shown, the data storage method of the exemplary embodiments of this disclosure may include:

[0053] Step 101: Divide the original data into multiple data fragments based on the data size fragment threshold, and each data size fragment threshold is less than the preset data capacity.

[0054] In an exemplary embodiment of this disclosure, when storing cached data, it is first determined whether the amount of the original data to be stored is greater than or equal to the preset data capacity. When the amount of the original data exceeds the preset data capacity, it is marked as shardable data, also known as large key data. When the amount of the original data does not exceed the data sharding threshold, it is marked as indivisible data, also known as small key data.

[0055] For example, the data capacity can be preset to 10KB. When the amount of raw data to be stored is greater than 10KB, the raw data is marked as shardable data. When the amount of raw data to be stored is less than 10KB, the raw data is marked as indivisible data.

[0056] For example, suppose the original data to be stored is 48KB, which is larger than the preset data capacity. This original data can be marked as shardable. In this case, when the sharding threshold is 8KB, since 48KB ÷ 8KB = 6, the original data can be exactly divided into 6 8KB shards. When the sharding threshold is 7KB, since 48KB ÷ 7KB ≈ 6.86, it cannot be exactly divided. That is, after splitting the original data into 6 shards, less than 1.0KB of original data remains. Therefore, the remaining original data can be used as a new shard, resulting in 7 shards. Thus, when the calculated number of shards based on the original data and the sharding threshold is a decimal, this decimal can be rounded up to obtain the actual number of shards.

[0057] In practical applications, as the amount of original data increases, the number of corresponding data shards may also increase. For example, assuming the data sharding threshold is 8KB, when the amount of original data A is 48KB, it can be split into 6 data shards, and when the amount of original data B is 56KB, it can be split into 7 data shards. That is, the amount of original data is positively correlated with the number of data shards, and the amount of data in each data shard is less than or equal to the data sharding threshold.

[0058] As can be seen, the exemplary embodiments of this disclosure can divide the original data into multiple data segments, which can not only effectively alleviate the storage problem of massive data and shorten the storage time, but also, as the amount of data continues to increase, each data segment can grow into new original data, thereby allowing for further segmentation and thus enabling rapid processing of massive data.

[0059] Step 102: Determine the ordering identifier of the original data based on the identifiers of multiple data fragments and the combination relationships of the multiple data fragments. The ordering identifier includes the identifiers of multiple data fragments sorted according to the combination relationships of the data fragments.

[0060] For example, after splitting the original data into multiple data fragments, there are corresponding combination relationships between these fragments. Specifically, the multiple fragments obtained from splitting the original data can be combined in an ordered manner to form a complete original data set. Based on this, the multiple data fragments can be identified, and by combining these identifications according to the combination relationships between them, a complete original data set can be obtained. This combination of identifications can be achieved by sorting the identifications or by marking them with an ordered sequence.

[0061] Figure 2 A flowchart illustrating a data fragmentation method according to an exemplary embodiment of this disclosure is shown. Figure 2 As shown, the original data 301 can be divided according to the data size and the multiple fragments can be identified in an ordered manner. The original data 301 is divided into 5 fragments: A, B, C, D and E, where A, B, C, D and E are the identifiers of the 5 fragments.

[0062] When the five fragments of the original data are labeled in the order A, B, C, D, and E, they can be combined to form the complete original data. Therefore, the ordered label can be {A, B, C, D, E}. When the five fragments of the original data are labeled in the order B, C, A, D, and E, the ordered label can be {B, C, A, D, E}. It is evident that the five fragments can be combined based on the ordered label to obtain the complete original data.

[0063] Step 103: Store the ordering identifier and multiple shard data on different machines, with each shard having its own corresponding machine for processing.

[0064] The machine in this exemplary embodiment can be a virtual cache machine or a physical machine. Different data can be stored in different cache machines within the same physical machine, or in different physical machines. If a cache machine has a set lifespan, it can be automatically reclaimed or destroyed when it reaches the end of its lifespan if no data is stored within it. For example, a 24-hour lifespan can be set for a machine cache; that is, if a cache exists on a machine for more than 24 hours, it can be automatically destroyed, thereby achieving hardware reuse and saving costs.

[0065] Exemplary embodiments of this disclosure can establish self-describing structures for both original data and fragmented data. The self-describing structure for the original data is determined based on an ordering identifier and the identifier of the original data; alternatively, the self-describing structure for each fragmented data can be determined. For example, fragmented data can be compressed, and then the compressed fragmented data can be stored in the data field of the self-describing structure of the fragmented data, with related information appended. The compression algorithm for the fragmented data can be a data stream compression algorithm, a lossless compression algorithm, or a compression library. Here, the self-describing structure can be referred to as semi-structured data or schema-less data, referring to a structured data form containing tags or tag elements. These tags or tag elements aim to separate semantic elements, generating a record and field hierarchy for given data. A self-describing structure is structured data composed of a batch of self-describing structures.

[0066] Figure 3 A schematic diagram illustrating a self-describing structure according to exemplary embodiments of the present disclosure is shown. Figure 3 As shown, the self-describing structure corresponding to the divisible raw data can include the key information of the raw data, the value information of the raw data, the ordering identifier groupKeyList, the version information Version, the mode information mode, the raw data identifier isBigKey, etc. The value information of the raw data does not store any data, and the raw data identifier isBigKey indicates that the raw data marked by the raw data identifier is big key data.

[0067] A self-describing structure for sharded data can include key information, value information, an ordering identifier (groupKeyList), version information, and mode information. The ordering identifier (groupKeyList) is empty. The value information actually stores the compressed sharded data, and the key information for different shards can be different.

[0068] For example, if there are five data shards, and the identifiers of the five data shards are sorted in the order of the first data shard, the second data shard, the third data shard, the fourth data shard, and the fifth data shard, and their corresponding key information is key1, key2, key3, key4, and key5 respectively, then the order identifier groupKeyList of the five data shards in the self-description structure of the original data to which they belong can be {key1, key2, key3, key4, key5}.

[0069] After data fragmentation, multiple data sets need to be stored. The self-describing structure of the original data can be stored in the storage space of the first machine, while the self-describing structures of the multiple fragments can be stored in the storage space of the second machine. The self-describing structures of the multiple fragments can be stored in parallel or serially in the storage spaces of different second machines.

[0070] In practical applications, it is possible to locate the first machine storing the self-describing structure of the original data and the second machine storing the self-describing structure of different shards in a machine cluster. For example, a consistent hash algorithm can be used to locate the machines storing the self-describing structure of the original data and the self-describing structures of different shards.

[0071] In one alternative embodiment, the exemplary embodiment of this disclosure may further include: in response to the failure of storing the self-description structure of the original data in the first storage space, indicating that the storage of the self-description structure of the original data has failed. At this time, although the first storage space has been allocated for the self-description structure of the original data in the first machine, it does not currently store the self-description structure of the original data. Therefore, the first storage space can be used to store other data, which can be various possible self-description structures, thus achieving secondary utilization of the first storage space. Simultaneously, to avoid data interference, a lifecycle can be set for the first storage space that has been created but not yet cached. If the first storage space reaches the end of its lifecycle, the first machine can automatically destroy the cache.

[0072] For example, when storing the original data self-describing structure, the disclosed example embodiment can determine whether there is a first storage space in the first machine to determine whether there is storage space that may be reused.

[0073] If a first storage space has already been created in the first machine, it means that there may be a storage space in the first machine that has been reused. In this case, it can be determined whether the data volume of the self-describing structure of the original data matches the capacity of the first storage space.

[0074] When the amount of data in the self-describing structure of the original data matches the capacity of the first storage space of the first machine, it means that the capacity of the first storage space of the first machine meets the storage requirements of the self-describing structure of the original data. The self-describing structure of the original data can be directly stored in the first storage space of the first machine, thereby achieving the purpose of reusing the first storage space of the first machine. When the amount of data in the self-describing structure of the original data does not match the capacity of the first storage space of the first machine, it means that the capacity of the first storage space of the first machine cannot meet the storage requirements of the self-describing structure of the original data. A second storage space of the first machine that matches the amount of data in the self-describing structure of the original data can be created in the first machine, and then the self-describing structure of the original data can be stored in the second storage space of the first machine.

[0075] If the first storage space of the first machine does not exist, it means that there is no storage space in the first machine that can be reused. In this case, a second storage space of the first machine needs to be created in the first machine that matches the data volume of the self-describing structure of the original data, and then the self-describing structure of the original data is stored in the second storage space of the first machine.

[0076] When creating a second storage space in the first machine, the second storage space can be established on the first machine based on the self-describing body structure of the original data, and then the self-describing body structure of the original data can be stored in the second storage space of the first machine.

[0077] In an alternative embodiment, this disclosure may further include: in response to a failure to store the self-describing structure of the fragmented data in the second storage space, indicating a storage failure of the self-describing structure of the fragmented data. At this time, although a second storage space has been allocated in the second machine for the self-describing structure of the fragmented data, this second storage space does not currently store the self-describing structure of the fragmented data. Therefore, the second storage space can be used to store other data, which may include various possible self-describing structures, thus achieving secondary utilization of the second storage space. Simultaneously, to avoid data interference, a lifecycle can be set for the created but uncached second storage space. If the second storage space reaches the end of its lifecycle, the second machine can automatically destroy the cache. It should be understood that the storage failure of the self-describing structure may be due to network communication failures, network jitter, or other problems.

[0078] For example, when storing fragmented data self-describing structures, the disclosed example embodiments can determine whether there is a first storage space of the second machine in the second machine, so as to determine whether there is storage space that may be reused.

[0079] If the second machine has an existing first storage space, it means that there may be storage space in the second machine that has been reused. In this case, it can be determined whether the data volume of the self-describing structure of the fragmented data matches the capacity of the first storage space of the second machine.

[0080] When the data size of the self-describing structure of the fragmented data matches the capacity of the first storage space of the second machine, it means that the capacity of the second storage space meets the storage requirements of the self-describing structure of the fragmented data. The self-describing structure of the fragmented data can be directly stored in the first storage space of the second machine, thereby achieving the purpose of reusing the second storage space. When the data size of the self-describing structure of the fragmented data does not match the capacity of the first storage space of the second machine, it means that the capacity of the first storage space of the second machine cannot meet the storage requirements of the self-describing structure of the fragmented data. A second storage space of the second machine can be created in the second machine that matches the data size of the self-describing structure of the fragmented data, and then the self-describing structure of the fragmented data can be stored in the second storage space of the second machine.

[0081] If the first storage space of the second machine does not exist in the second machine, it means that there is no storage space in the second machine that can be reused. In this case, a second storage space of the second machine needs to be created in the second machine to match the data volume of the self-description structure of the fragmented data, and then the self-description structure of the fragmented data is stored in the second storage space of the second machine.

[0082] When creating a second storage space for the second machine, the second storage space can be established on the second machine based on the self-describing structure of the fragmented data, and then the self-describing structure of the fragmented data can be stored in the second storage space of the second machine.

[0083] The exemplary embodiments of this disclosure can also adaptively select whether to shard the original data. If the original data is not sharded, it can be stored directly. If the original data needs to be sharded, the data storage method of the embodiments of this disclosure is executed to shard the original data and then store it. Moreover, the splitting is not affected by changes in business. As the business volume increases in the later period, that is, as the data volume increases, the sharded data can also adaptively grow into new original data without manual intervention.

[0084] As can be seen, storing fragmented data on different machines allows a large original dataset to be split into multiple fragments. By storing these fragments on different machines, storage can be performed more quickly, and all storage machines can be reused, significantly reducing costs.

[0085] Figure 4A basic flowchart of a data reading method according to an exemplary embodiment of this disclosure is shown. Figure 4 As shown, the data reading method of an exemplary embodiment of this disclosure may include:

[0086] Step 401: In response to the data retrieval request, obtain an ordering identifier matching the data retrieval request from the distributed storage machine. The ordering identifier is used to indicate the combination relationship of different data shards and that different data shards are stored in different machines. The data volume of each data shard is less than the preset data capacity.

[0087] For example, in the exemplary embodiments of this disclosure, when reading data, a data retrieval request can be parsed to obtain query keywords. If there are multiple query keywords, the logical relationship between different query keywords can also be parsed from the data retrieval request. Based on this, the query key information is determined based on the query keywords or the query keywords and logical relationships. Then, the target key information matching the query key information is determined from the key information of the original data in the self-description structure of a large amount of original data. It should be understood that when the data corresponding to the query key information is stored in a local cache, the data can be retrieved directly from the local cache without needing to be read through an external machine.

[0088] The self-description structure of the original data containing the target key information (key) in this exemplary embodiment is actually the self-description structure of the data to be retrieved. If the original data identifier contained in the self-description structure is the original data identifier isBigKey, it indicates that the data is big key data, the value information of the original data contained in the self-description structure is empty, and the ordering identifier groupKeyList contained therein matches the data retrieval request.

[0089] Step 402: Combine the fragmented data stored in different machines based on the ordering identifier to obtain the data reading result.

[0090] The ordering identifier of the exemplary embodiments of this disclosure can be used to indicate the combination relationship of different fragment data. It includes multiple ordering identifiers, which include the identifiers of multiple fragment data ordered according to the combination relationship of fragment data. Therefore, fragment data can be obtained from the corresponding machine based on the identifier of each fragment data, and the fragment data corresponding to the multiple fragment data storage identifiers can be stored according to the combination relationship of fragment data to obtain the data reading result.

[0091] The exemplary embodiments disclosed herein can also implement a multi-grayscale strategy to achieve fault-free online deployment. For example, a prefix identifier can be added to a certain part of the data. Data with the prefix identifier will automatically match the scheme of the embodiments disclosed herein, while data without the prefix identifier will still be processed in the previous way.

[0092] For example, when storing data, the data corresponding to each data node can be monitored and a whitelist can be established. The user data in the whitelist is data that supports the gray-scale strategy. For example, a prefix can be added to the key information corresponding to the original data that needs to execute the rational method of this disclosure, such as prefix+key. Then, all original data with the key information prefix+key will execute the method of the exemplary embodiment of this disclosure, while other key information will execute other schemes.

[0093] The multi-level grayscale strategy can also include splitting multiple raw data streams based on user address, user identification code, or user-used machine information. For example, when splitting multiple raw data streams based on user address A, the method of the exemplary embodiment of this disclosure can be directly executed for the raw data with address information A, while other methods are executed for the other raw data with address information, thereby splitting the data streams and making data processing faster. Furthermore, based on multiple splitting strategies, online gradual grayscale adjustments can be made to ensure a smooth upgrade upon deployment.

[0094] The exemplary embodiments disclosed herein are fully compatible with the original method in terms of external use, and can shield implementation details. In the process of modifying the original system, only a few lines of code need to be changed. Furthermore, through data processing logic optimization, this disclosure does not require upgrading the existing cache component hardware. Therefore, it not only saves costs, but also has high compatibility in terms of deployment method and is simple to use.

[0095] One or more technical solutions provided in the exemplary embodiments of this disclosure can shard the original data based on a data volume sharding threshold, thereby obtaining multiple shards of the original data, where each data volume sharding threshold is less than a preset data capacity. Based on this, when storing data, for original data exceeding the preset data capacity, the original data can be split according to the data volume scale to obtain multiple shards. Therefore, as the business volume increases, the data volume of the shards will not change. In this case, storing multiple shards on different machines reduces the risk of insufficient machine caching and network congestion, thereby improving data storage efficiency. Furthermore, it eliminates the need for additional high-performance hardware and manual intervention, thus reducing costs.

[0096] Furthermore, the exemplary embodiments of this disclosure can also determine the ordering identifier of the original data based on the storage addresses of multiple data shards and the combination relationship of the multiple data shards, and store the ordering identifier in different machines. Based on this, in response to a data retrieval request, the ordering identifier matching the data retrieval request can be obtained from the distributed storage machine, and the data shards stored in different machines can be combined based on the ordering identifier to obtain the final data retrieval result. It is evident that the exemplary embodiments of this disclosure, when storing multiple data shards, do not require reconstruction of the online system; only data processing is performed, thereby reducing deployment risks and minimizing business losses caused by online failures.

[0097] The foregoing primarily describes the solutions provided by the embodiments of this disclosure from the perspective of the server. It is understood that, in order to implement the above functions, the server includes the corresponding hardware structures and / or software modules for executing each function. Those skilled in the art should readily recognize that, based on the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein, this disclosure can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this disclosure.

[0098] This disclosure embodiment can divide the server into functional units according to the above method example. For example, it can divide each function into a separate functional module, or it can integrate two or more functions into one processing module. The integrated module can be implemented in hardware or as a software functional module. It should be noted that the module division in this disclosure embodiment is illustrative and only represents one logical functional division. In actual implementation, there may be other division methods.

[0099] By dividing each functional module according to its corresponding function, an exemplary embodiment of this disclosure provides a data storage device, which can be a server or a chip applied to a server. Figure 5 A schematic block diagram of the functional modules of a data storage device according to an exemplary embodiment of the present disclosure is shown. Figure 5 As shown, the data storage device 500 includes:

[0100] The sharding module 501 is used to shard the original data based on a data volume sharding threshold to obtain multiple shards of the original data, wherein the data volume sharding threshold is less than a preset data capacity.

[0101] The determining module 502 is used to determine an ordering identifier based on the identifiers of multiple data fragments and the combination relationship of the multiple data fragments;

[0102] Storage module 503 is used to store the ordering identifier and multiple fragmented data in different machines.

[0103] In one possible implementation, the amount of the original data is positively correlated with the number of the fragmented data, and the amount of the fragmented data is less than or equal to the data fragmentation threshold.

[0104] In one possible implementation, the ordering identifier includes identifiers of multiple data fragments ordered according to the combination relationship of the fragment data.

[0105] In one possible implementation, the storage module 503 is further configured to determine the self-describing structure of the original data based on the ordering identifier and the identifier of the original data; determine the self-describing structure of the fragmented data based on each fragmented data; store the self-describing structure of the original data in the storage space of the first machine; and store the self-describing structures of multiple fragmented data in the storage space of the second machine.

[0106] In one possible implementation, if a first storage space for the first machine already exists in the first machine, determine that the data size of the self-describing structure of the original data matches the capacity of the first storage space of the first machine; if a first storage space for the first machine already exists in the first machine, and if the capacity of the self-describing structure of the original data does not match the capacity of the first storage space of the first machine, create a second storage space for the first machine in the first machine; if a first storage space for the first machine already exists in the first machine, create a second storage space in the first machine that matches the data size of the self-describing structure of the original data.

[0107] In one possible implementation, in response to the failure to store the self-describing structure of the original data in the first storage space, a message indicating that the storage of the self-describing structure of the original data has failed is displayed.

[0108] In one possible implementation, if a first storage space of the second machine already exists in the second machine, determine that the data size of the self-describing structure of the fragmented data matches the capacity of the first storage space of the second machine; if a first storage space of the second machine already exists in the second machine, and if the capacity of the self-describing structure of the fragmented data does not match the capacity of the first storage space of the second machine, create a second storage space of the second machine in the second machine; if a first storage space of the second machine already exists in the second machine, create a second storage space in the second machine that matches the data size of the self-describing structure of the fragmented data.

[0109] In one possible implementation, in response to the failure to store the self-describing structure of the fragmented data in the first storage space, a message indicating that the self-describing structure of the fragmented data has failed to be stored is displayed.

[0110] By dividing each functional module according to its corresponding function, an exemplary embodiment of this disclosure provides a data reading device, which can be a server or a chip applied to a server. Figure 6 A schematic block diagram of the functional modules of a data reading apparatus according to an exemplary embodiment of the present disclosure is shown. Figure 6 As shown, the data reading device 600 includes:

[0111] The response module 601 is used to respond to a data retrieval request by obtaining an ordering identifier matching the data retrieval request from a distributed storage machine. The ordering identifier is used to indicate the combination relationship of different data shards. The ordering identifier and the different data shards are stored in different machines, and the data volume of each data shard is less than a preset data capacity.

[0112] The acquisition module 602 is used to combine the fragmented data stored in different machines based on the ordering identifier to obtain the data reading result.

[0113] In one possible implementation, the acquisition module 602 is further configured to include the ordering identifier as a combination relationship of fragmented data and the identifiers of multiple fragmented data. The step of combining the fragmented data stored in different machines based on the ordering identifier to obtain a data reading result includes: acquiring the fragmented data from the corresponding machine based on the identifier of each fragmented data; and obtaining the data reading result by combining the fragmented data corresponding to the multiple fragmented data storage identifiers according to the combination relationship of the fragmented data.

[0114] Figure 7 A schematic block diagram of a chip according to an exemplary embodiment of the present disclosure is shown. Figure 7As shown, the chip 700 includes one or more (including two) processors 701 and a communication interface 702. The communication interface 702 can support the server in performing the data transmission and reception steps in the above method, and the processor 701 can support the server in performing the data processing steps in the above method.

[0115] Optional, such as Figure 7 As shown, the chip 700 also includes a memory 703, which may include read-only memory and random access memory, and provides operation instructions and data to the processor. A portion of the memory may also include non-volatile random access memory (NVRAM).

[0116] In some implementations, such as Figure 7 As shown, processor 701 executes corresponding operations by calling operation instructions stored in memory (which may be stored in the operating system). Processor 701 controls the processing operations of any terminal device; processor can also be called a central processing unit (CPU). Memory 703 may include read-only memory and random access memory, and provides instructions and data to processor 701. A portion of memory 703 may also include NVRAM. For example, in applications, memory, communication interfaces, and other components are coupled together via a bus system, which may include, in addition to a data bus, a power bus, a control bus, and a status signal bus, etc. However, for clarity, in... Figure 7 The general designated all buses as Bus System 704.

[0117] The methods disclosed in the embodiments of this disclosure can be applied to a processor or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above methods can be completed by integrated logic circuits in the processor's hardware or by instructions in software form. The processor can be a general-purpose processor, a digital signal processor (DSP), an ASIC, a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this disclosure. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this disclosure can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules can reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. This storage medium is located in memory; the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above methods.

[0118] Exemplary embodiments of this disclosure also provide an electronic device, including: at least one processor; and a memory communicatively connected to the at least one processor. The memory stores a computer program executable by the at least one processor, the computer program being executed by the at least one processor to cause the electronic device to perform a method according to an embodiment of this disclosure.

[0119] Exemplary embodiments of this disclosure also provide a non-transitory computer-readable storage medium storing a computer program, wherein the computer program, when executed by a computer's processor, is used to cause the computer to perform a method according to embodiments of this disclosure.

[0120] Exemplary embodiments of this disclosure also provide a computer program product, including a computer program, wherein, when executed by a processor of a computer, the computer program is used to cause the computer to perform a method according to an embodiment of this disclosure.

[0121] refer to Figure 8The present invention describes a structural block diagram of an electronic device that can serve as a server or client of the present disclosure, which is an example of a hardware device that can be applied to various aspects of the present disclosure. The electronic device is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0122] like Figure 8 As shown, the electronic device 800 includes a computing unit 801, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 802 or a computer program loaded from a storage unit 808 into a random access memory (RAM) 803. The RAM 803 may also store various programs and data required for the operation of the device 800. The computing unit 801, ROM 802, and RAM 803 are interconnected via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0123] Multiple components in electronic device 800 are connected to I / O interface 805, including: input unit 806, output unit 807, storage unit 808, and communication unit 809. Input unit 806 can be any type of device capable of inputting information to electronic device 800. Input unit 806 can receive input digital or character information and generate key signal inputs related to user settings and / or function control of electronic device. Output unit 807 can be any type of device capable of presenting information and may include, but is not limited to, a display, speaker, video / audio output terminal, vibrator, and / or printer. Storage unit 808 may include, but is not limited to, disks and optical discs. Communication unit 809 allows electronic device 800 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks, and may include, but is not limited to, modems, network cards, infrared communication devices, wireless communication transceivers, and / or chipsets, such as Bluetooth™ devices, WiFi devices, WiMax devices, cellular communication devices, and / or the like.

[0124] The computing unit 801 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 801 performs the various methods and processes described above. For example, in some embodiments, the methods of exemplary embodiments of this disclosure can be implemented as computer software programs tangibly contained in a machine-readable medium, such as storage unit 808. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 800 via ROM 802 and / or communication unit 809. In some embodiments, the computing unit 801 can be configured to perform the methods of exemplary embodiments of this disclosure by any other suitable means (e.g., by means of firmware).

[0125] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0126] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0127] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0128] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0129] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other.

[0130] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer programs or instructions. When the computer program or instructions are loaded and executed on a computer, the processes or functions described in the embodiments of this disclosure are performed, in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, a terminal, a user equipment, or other programmable device. The computer program or instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another. For example, the computer program or instructions can be transferred from one website, computer, server, or data center to another website, computer, server, or data center via wired or wireless means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center integrating one or more available media. The available medium can be a magnetic medium, such as a floppy disk, hard disk, or magnetic tape; it can also be an optical medium, such as a digital video disc (DVD); or it can be a semiconductor medium, such as a solid-state drive (SSD).

[0131] Although this disclosure has been described in conjunction with specific features and embodiments, it will be apparent that various modifications and combinations can be made therein without departing from the spirit and scope of this disclosure. Accordingly, this specification and drawings are merely exemplary illustrations of the disclosure as defined by the appended claims and are to be considered as covering any and all modifications, variations, combinations, or equivalents within the scope of this disclosure. It is obvious that those skilled in the art can make various alterations and modifications to this disclosure without departing from its spirit and scope. Thus, this disclosure is also intended to include any such modifications and modifications that fall within the scope of the claims of this disclosure and their equivalents.

Claims

1. A data storage method, characterized in that, The method includes: The original data is divided into multiple fragments based on a data volume fragmentation threshold, wherein the data volume fragmentation threshold is less than a preset data capacity. Based on the identifiers of multiple data fragments and the combination relationships of the multiple data fragments, an ordering identifier is determined; The ordering identifier and the multiple fragmented data are stored on different machines; The step of storing the ordering identifier and multiple fragmented data in different machines includes: The self-describing structure of the original data is determined based on the ordering identifier and the identifier of the original data; A self-describing structure for each of the data segments is determined based on the data segment itself. The self-describing structure of the original data is stored in the storage space of the first machine; The self-describing structures of the multiple fragmented data are stored in the storage space of the second machine.

2. The method according to claim 1, characterized in that, The amount of the original data is positively correlated with the number of the fragmented data, and the amount of the fragmented data is less than or equal to the data fragmentation threshold.

3. The method according to claim 1, characterized in that, The ordering identifier includes the identifiers of multiple data fragments sorted according to the combination relationship of the fragment data.

4. The method according to claim 1, characterized in that, The method further includes: If a first storage space of the first machine already exists in the first machine, determine that the data volume of the self-describing structure of the original data matches the capacity of the first storage space of the first machine; If a first storage space for the first machine already exists in the first machine, and if the capacity of the self-describing structure of the original data does not match the capacity of the first storage space of the first machine, a second storage space for the first machine is created in the first machine. If the first storage space of the first machine does not exist in the first machine, a second storage space matching the data volume of the self-describing structure of the original data is created in the first machine.

5. The method according to claim 4, characterized in that, The method further includes: In response to the failure to store the self-describing structure of the original data in the first storage space, a message indicating that the storage of the self-describing structure of the original data failed is displayed.

6. The method according to claim 1, characterized in that, The method further includes: If the second machine has an existing first storage space, determine that the data volume of the self-describing structure of the fragmented data matches the capacity of the first storage space of the second machine. If a first storage space of the second machine already exists in the second machine, and if the self-describing structure of the fragmented data does not match the capacity of the first storage space of the second machine, a second storage space of the second machine is created in the second machine. If the first storage space of the second machine does not exist in the second machine, a second storage space matching the data volume of the self-describing structure of the fragmented data is created in the second machine.

7. The method according to claim 6, characterized in that, The method further includes: In response to the failure to store the self-describing structure of the fragmented data in the first storage space, a message indicating that the self-describing structure of the fragmented data has failed to be stored is displayed.

8. A data reading method for reading data stored by the data storage method as described in claim 1, characterized in that, include: In response to a data retrieval request, an ordering identifier matching the data retrieval request is obtained from the distributed storage machine based on the data retrieval request. The ordering identifier is used to indicate the combination relationship of different data shards. The ordering identifier and the different data shards are stored in different machines, and the data volume of each data shard is less than a preset data capacity. Based on the ordered identifier, the fragmented data stored in different machines are combined to obtain the data reading result.

9. The method according to claim 8, characterized in that... Therefore, the ordering identifier includes the combination relationship of the fragmented data and the identifiers of multiple fragmented data. The step of combining the fragmented data stored in different machines based on the ordering identifier to obtain the data reading result includes: The fragment data is obtained from the corresponding machine based on the identifier of each fragment data; The data reading result is obtained by combining the fragmented data according to the fragmented data combination relationship of multiple fragmented data storage identifiers.

10. A data storage device, characterized in that, The device includes: The sharding module is used to shard the original data based on a data volume sharding threshold to obtain multiple shards of the original data, wherein the data volume sharding threshold is less than a preset data capacity. The determination module is used to determine the ordering identifier based on the identifiers of multiple data fragments and the combination relationship of the multiple data fragments; A storage module is used to store the ordering identifier and multiple fragmented data in different machines; The storage module is further configured to determine the self-describing structure of the original data based on the ordering identifier and the identifier of the original data; determine the self-describing structure of the fragmented data based on each fragmented data; store the self-describing structure of the original data in the storage space of the first machine; and store the self-describing structures of multiple fragmented data in the storage space of the second machine.

11. A data reading device for reading data stored in the data storage device as described in claim 10, characterized in that, The device includes: A response module is used to respond to a data retrieval request by obtaining an ordering identifier matching the data retrieval request from a distributed storage machine. The ordering identifier is used to indicate the combination relationship of different data shards. The ordering identifier and the different data shards are stored in different machines, and the data volume of each data shard is less than a preset data capacity. The acquisition module is used to combine the fragmented data stored in different machines based on the ordered identifier to obtain the data reading result.

12. An electronic device, characterized in that, include: processor; And, the memory for storing programs; The program includes instructions that, when executed by the processor, cause the processor to perform the method according to any one of claims 1 to 9.

13. A non-transitory computer-readable storage medium, characterized in that, The non-transitory computer-readable storage medium stores computer instructions for causing the computer to perform the method according to any one of claims 1 to 9.

Images