Data processing method and device and electronic equipment

Abstract

The embodiment of the invention provides a data processing method and device and electronic equipment, and relates to the technical field of data storage.The method comprises the steps that all pieces of target data are determined from data stored in a target storage system; determining a target fragment in a hash ring set by the target storage system based on each piece of target data; obtaining an adjustment strategy for the target fragment; and executing the adjustment strategy on the target fragment to adjust the target fragment according to the adjustment mode, and respectively mapping the at least one piece of target data to respective corresponding target nodes. Therefore, by means of the scheme, the situation that loads of the distributed storage system are not uniform can be relieved in time.

Description

Technical Field

[0001] This application relates to the field of data storage technology, and in particular to a data processing method, apparatus, and electronic device. Background Technology

[0002] In a distributed storage system, data is scattered across multiple storage nodes. Access requests need to be precisely routed to the corresponding storage node, which is the node storing the data to be accessed. Currently, a hash ring approach is used, employing a circular hash space to achieve efficient binding between data, access requests, and nodes (storage nodes or their corresponding virtual nodes). Specifically, each node is mapped to a point on the hash ring via its hash value; each piece of data is mapped to a point on the hash ring via its hash value. Starting from that point, a clockwise or counter-clockwise search is performed to find the nearest node, thus determining which node is responsible for storing or processing the data. This network of nodes divides the entire hash ring into multiple shards.

[0003] When encountering sudden, high-frequency access to specific data, or when the data's popularity shifts from hot to cold, uneven load distribution can occur in the distributed storage system. Related technologies require manual intervention to adjust the target shards mapped to the data on the hash ring, ensuring the data is mapped to the adjusted shards. However, manual intervention has a drawback of latency, meaning uneven load distribution can still persist in the distributed storage system. Summary of the Invention

[0004] The purpose of this application is to provide a data processing method, apparatus, and electronic device to promptly alleviate the uneven load situation in distributed storage systems. The specific technical solution is as follows:

[0005] In a first aspect provided by the embodiments of this application, a data processing method is first provided, the method comprising:

[0006] From the data stored in the target storage system, each target data is determined; wherein each target data is data that meets predetermined data prediction conditions; the predetermined data prediction conditions include a first prediction condition for predicting hot data, or a second prediction condition for predicting cold data;

[0007] Based on the target data, the target fragment in the hash ring set by the target storage system is determined; wherein, the target fragment is a fragment used to map at least one target data.

[0008] Obtain the adjustment strategy for the target shard; wherein the adjustment strategy is used to indicate the adjustment method of the target shard and the target node corresponding to the shard obtained after adjustment;

[0009] The adjustment strategy is executed on the target shard to adjust the target shard according to the adjustment method, and the at least one target data is mapped to its corresponding target node; wherein, the target node corresponding to each target data is the target node corresponding to the shard to which the target data needs to be mapped.

[0010] In a second aspect provided in the embodiments of this application, a data processing apparatus is also provided, the apparatus comprising:

[0011] The first determining module is used to determine each piece of target data from the data stored in the target storage system; wherein each piece of target data is data that meets predetermined data prediction conditions; the predetermined data prediction conditions include a first prediction condition for predicting hot data, or a second prediction condition for predicting cold data.

[0012] The second determining module is used to determine the target fragment in the hash ring set by the target storage system based on each target data; wherein the target fragment is a fragment used to map at least one target data.

[0013] An acquisition module is used to acquire an adjustment strategy for the target shard; wherein the adjustment strategy is used to indicate the adjustment method of the target shard and the target node corresponding to the shard obtained after adjustment;

[0014] An execution module is used to execute the adjustment strategy on the target shard to adjust the target shard according to the adjustment method, and to map the at least one target data to its corresponding target node; wherein, the target node corresponding to each target data is the target node corresponding to the shard to which the target data needs to be mapped.

[0015] In a third aspect provided in the embodiments of this application, an electronic device is also provided, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; the memory is used to store computer programs; and the processor is used to implement any of the data processing methods provided in the first aspect when executing the program stored in the memory.

[0016] In another aspect provided by the embodiments of this application, a computer-readable storage medium is also provided, wherein a computer program is stored therein, and when the computer program is executed by a processor, it implements any of the data processing methods provided in the first aspect above.

[0017] In another aspect provided by the embodiments of this application, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to perform any of the data processing methods provided in the first aspect above.

[0018] As can be seen from the above, the data processing method provided in this application provides a method for predicting target data in a future time period and obtaining an adjustment strategy for the target shard used to map at least one target data for the predicted target data. Thus, when encountering a sudden high-frequency access to a certain data or when the data popularity of a certain data changes from hot data to cold data, the target shard can be directly adjusted according to the obtained adjustment strategy to alleviate the uneven load situation in the distributed storage system in a timely manner. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below.

[0020] Figure 1 A flowchart illustrating a data processing method provided in an embodiment of this application;

[0021] Figure 2 A flowchart illustrating a predetermined first analysis method provided for an embodiment of this application;

[0022] Figure 3 A flowchart illustrating another predetermined first analysis method provided for an embodiment of this application;

[0023] Figure 4 A flowchart illustrating a predetermined second analysis method provided for an embodiment of this application;

[0024] Figure 5 A flowchart illustrating another predetermined second analysis method provided for an embodiment of this application;

[0025] Figure 6 A schematic diagram of a multi-layer hash ring provided in an embodiment of this application;

[0026] Figure 7 This application provides a schematic diagram of the architecture of a specific embodiment.

[0027] Figure 8 This is a schematic diagram of the structure of a data processing device provided in an embodiment of this application;

[0028] Figure 9 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0029] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art based on this application are within the scope of protection of this application.

[0030] To address the aforementioned technical problems, embodiments of this application provide a data processing method, apparatus, and electronic device. This method is applicable to various application scenarios involving sharding adjustments, such as sharding hash rings or merging hash ring shards. Furthermore, this method can be applied to various electronic devices, including laptops and desktop computers (hereinafter referred to as electronic devices). Therefore, embodiments of this application do not limit the application scenarios or the executing entity of this method.

[0031] This application provides a data processing method that may include the following steps:

[0032] From the data stored in the target storage system, each target data is determined; wherein each target data is data that meets predetermined data prediction conditions; the predetermined data prediction conditions include a first prediction condition for predicting hot data, or a second prediction condition for predicting cold data;

[0033] Based on the target data, the target shards on the hash ring set by the target storage system are determined; wherein, the target shards are shards used to map at least one target data.

[0034] Obtain the adjustment strategy for the target shard; wherein the adjustment strategy is used to indicate the adjustment method of the target shard and the target node corresponding to the shard obtained after adjustment;

[0035] The adjustment strategy is executed on the target shard to adjust the target shard according to the adjustment method, and the at least one target data is mapped to its corresponding target node; wherein the target node corresponding to each target data is the target node corresponding to the shard to which the target data needs to be mapped.

[0036] As can be seen from the above, the data processing method provided in this application provides a method for predicting target data in a future time period and obtaining an adjustment strategy for the target shard used to map at least one target data for the predicted target data. Thus, when encountering a sudden high-frequency access to a certain data or when the data popularity of a certain data changes from hot data to cold data, the target shard can be directly adjusted according to the obtained adjustment strategy to alleviate the uneven load situation in the distributed storage system in a timely manner.

[0037] Figure 1 This is a flowchart illustrating a data processing method provided in an embodiment of this application, as shown below. Figure 1 As shown, the method may include the following steps:

[0038] S101: Determine each piece of target data from the data stored in the target storage system.

[0039] Each target data is data that meets predetermined data prediction conditions; the predetermined data prediction conditions include a first prediction condition for predicting hot data, or a second prediction condition for predicting cold data.

[0040] In specific examples, data stored in a storage system can be categorized into hot data and cold data based on its frequency of access. Hot data refers to data that is frequently accessed, modified, or requires rapid response within a short period (e.g., 24 hours, 48 ​​hours); cold data refers to data that is accessed very infrequently or even infrequently for extended periods (e.g., months, years). It's important to note that frequently accessed hot data in a storage system does not necessarily mean it is suddenly accessed within a very short timeframe. Before becoming frequently accessed, the access volume of this type of data tends to increase. Conversely, cold data in a storage system does not necessarily mean it is completely inaccessible for a very short period. Before transitioning from hot data to cold data, the access volume of this type of data tends to decrease.

[0041] However, when encountering a sudden surge in high-frequency access to certain data (hereinafter referred to as the first type of event), or when the data's popularity shifts from hot to cold (hereinafter referred to as the second type of event), uneven load distribution may occur in the distributed storage system. For example, encountering the first type of event may cause the target shard used to map the hot data to become overloaded, resulting in uneven load distribution in the storage system and even the risk of a cascading failure. For instance, during the first type of event, the instantaneous access volume to the target shard used to map the hot data may far exceed the threshold that the target shard can handle, potentially causing it to become overloaded. In this case, high-frequency access requests concentrate on consuming the I / O (Input / Output) resources of the target shard, while other shards, due to a lack of access requests, have idle I / O resources, ultimately leading to uneven load distribution in the storage system. Similarly, encountering the second type of event may cause shards to become idle, also resulting in uneven load distribution in the storage system and even potential resource waste.

[0042] Based on this, this application pre-sets predetermined data prediction conditions for the above two types of events. These data prediction conditions include: a first prediction condition for predicting hot data for the first type of event, or a second prediction condition for predicting cold data for the second type of event.

[0043] Thus, based on the first prediction condition, each piece of data that meets the first prediction condition and belongs to the predicted hot data is identified from the data stored in the target storage system and is used as each piece of target data. Alternatively, based on the second prediction condition, each piece of data that meets the second prediction condition and belongs to the predicted cold data is identified from the data stored in the target storage system and is used as each piece of target data, thereby achieving the prediction of data that may lead to the aforementioned first or second type of events in the future.

[0044] S102: Based on each target data, determine the target fragment in the hash ring set by the target storage system.

[0045] The target shard is a shard used to map at least one piece of target data.

[0046] In this application, based on the predicted target data, a shard for mapping at least one target data is determined within the hash ring of the target storage system, and this shard is designated as the target shard. Specifically, each target data is mapped to a point on the hash ring of the target storage system through a hash value. Then, starting from that point, a search is performed clockwise or counterclockwise to find the nearest node, which is then selected as the node capable of processing the target data. Thus, the shard corresponding to that node is determined as the target shard for mapping the target data.

[0047] Optionally, the hash ring set in the target storage system is a multi-layered hash ring design. That is, the target storage system sets up a region-level global hash ring and at least one node-level region hash ring, with each region hash ring divided into multiple shards by at least one node. It should be noted that the data range that each node in the global hash ring can map is larger than the data range that each node in the region hash ring can map, and the data range that each node in the region hash ring can map is larger than the data range that each shard can map. Thus, when determining the target shard for each piece of target data, firstly, the hash value is mapped to the global hash ring to determine the region hash ring whose mapped range contains that hash value. Then, from the region hash ring, the shards whose mapped range contains that hash value are determined to obtain the target shard used to map the target data.

[0048] S103: Obtain the adjustment strategy for the target fragment.

[0049] The adjustment strategy is used to indicate how the target shard is adjusted and the target node corresponding to the shard obtained after the adjustment.

[0050] The purpose of this application is to adjust the shards used to map the predicted target data in a timely manner when encountering the first type of event or the second type of event mentioned above, by predicting the target data in the future period, so as to alleviate the uneven load situation in the storage system in a timely manner.

[0051] Therefore, after determining the target shard, the adjustment method for instructing the target shard and the target node corresponding to the shard obtained after adjustment are obtained. Thus, when encountering the first type of event or the second type of event mentioned above, the target shard can be directly adjusted according to the obtained adjustment strategy to reduce the time required for shard adjustment and to achieve timely relief of the uneven load situation in the storage system.

[0052] Specifically, for the first type of event mentioned above (encountering sudden high-frequency access to hot data), the identified target data is data that meets the first prediction condition, that is, the target data is predicted hot data among the data stored in the storage system that has the potential to become hot data in the future. Correspondingly, after determining the target shard, the obtained adjustment strategy is specifically used to indicate the splitting method of the target shard and the splitting strategy of the target node corresponding to each split shard. In this way, when encountering the first type of event mentioned above, the target shard can be directly split according to the obtained splitting strategy to reduce the time consumed by sharding processing, realize the timely distribution of access requests, and thus promptly alleviate the uneven load situation in the storage system.

[0053] It should be noted that after the target shard is sharded according to the obtained splitting strategy, the data that the target shard originally needed to map is split into each of the split shards. In other words, the data that all the split shards need to map are the same data that the target shard originally needed to map. After splitting the target shard, the target shard no longer exists. For example, the data that the target shard originally needed to map are data 1, data 2, data 3, data 4, and data 5; after splitting the target shard, split shard a needs to map data 1 and data 2, and split shard b needs to map data 3, data 4, and data 5.

[0054] For the second type of event mentioned above (the data popularity changes from hot data to cold data), the target data is determined to be data that meets the second prediction condition. That is, the target data is predicted cold data among the data stored in the storage system that has the potential to become cold data in the future. Correspondingly, after determining the target shard, the obtained adjustment strategy specifically indicates the required merging method for the target shard and the merging strategy for the target node corresponding to the merged shard. Thus, when encountering the second type of event, the target shard can be directly merged according to the obtained merging strategy. This reduces the number of shards, decreases the resource consumption of idle shards in the storage system, and thus promptly alleviates the uneven load on the storage system.

[0055] It should be noted that after merging the target fragment according to the obtained merging strategy, the data that the target fragment originally needed to map is merged into the merged fragment, which is responsible for the mapping. In other words, after merging the target fragment, the target fragment no longer exists; what exists is the merged fragment obtained from the merging. For example, if the target fragment originally needed to map data 1 and data 2, after merging the target fragment, the merged fragment is responsible for mapping data 1 and data 2.

[0056] Optionally, a default adjustment strategy is pre-set. Executing this default strategy adjusts the target shard according to the default adjustment method, and the target node corresponding to the resulting shard is also the default node. Thus, when adjustment of the target shard is needed, the default adjustment strategy can be directly executed, improving the timeliness of shard adjustment and further enhancing the timeliness of mitigating uneven load distribution in the storage system.

[0057] Optionally, for each target shard, an adjustment strategy is generated for that target shard to achieve targeted adjustment and improve the rationality of shard adjustment.

[0058] S104: Execute an adjustment strategy on the target shard to adjust the target shard according to the adjustment method, and map at least one target data to its corresponding target node.

[0059] The target node corresponding to each target data is the target node corresponding to the shard to which the target data needs to be mapped.

[0060] In this application, after obtaining the adjustment strategy for the target shard, the corresponding adjustment strategy can be executed on the target shard to adjust the target shard according to the adjustment method indicated by the adjustment strategy, and each target data in at least one target data is mapped to the target node corresponding to the shard to which the target data needs to be mapped, thereby realizing the adjustment of the target shard.

[0061] Optionally, in one embodiment, before performing the adjustment strategy on the target fragment in step S104 above, the data processing method provided in this application embodiment further includes the following step A:

[0062] Step A: Monitor whether the target shard meets the adjustment trigger conditions; if the adjustment trigger conditions are met, trigger the step of executing the adjustment strategy on the target shard.

[0063] The adjustment trigger condition is used to characterize: the changing trend of access volume to the target segment within a predetermined time period, which matches the changing trend set for the target condition; the target condition is the prediction condition included in the data prediction condition that is satisfied by at least one target data.

[0064] In this embodiment, considering the characteristics of frequently accessed "hot" data in the storage system, where the access volume increases before frequent access, and the characteristics of any record in the storage system, where the access volume decreases before it is converted to "cold" data, adjustment trigger conditions are pre-set. These adjustment trigger conditions characterize the trend of access volume to the target shard within a predetermined time period, matching the trend set for a target condition. The target condition is a prediction condition included in the data prediction conditions, satisfied by at least one piece of target data. Therefore, based on the set adjustment trigger conditions, it is determined whether an adjustment strategy needs to be implemented for the target shard. Specifically:

[0065] If, within a predetermined timeframe, the trend in access volume to a target shard matches the trend set by the first prediction condition included in the data prediction criteria for at least one piece of target data—that is, if the access volume to the target shard shows an increasing trend, and this increasing trend conforms to the increasing trend set by the first prediction condition—then it indicates a high probability that the target shard will become a hotspot shard in the future. In this case, to promptly distribute access requests during sudden, high-frequency access to hotspot data, a step of implementing an adjustment strategy for the target shard can be triggered to adjust the target shard in a timely manner.

[0066] If, within a predetermined timeframe, the trend in access volume to the target shard matches the trend set by the second prediction condition included in the data prediction criteria for at least one piece of target data—that is, the access volume to the target shard shows a downward trend, and this downward trend conforms to the downward trend set by the second prediction condition—then it indicates that the target shard is highly likely to become a cold shard in the future. In this case, to promptly reduce the number of idle shards in the storage system when the data popularity of a certain piece of data mapped by the target shard changes from hot data to cold data, thereby reducing the resource waste caused by idle shards, a step of implementing an adjustment strategy for the target shard can be triggered to adjust the target shard in a timely manner.

[0067] If, within a predetermined time period, the trend of access volume to the target segment does not match the trend set for the target condition—for example, the access volume of the target segment shows an increasing trend, but this increasing trend does not conform to the increasing trend set for the first prediction condition; or, the access volume of the target segment shows a decreasing trend, but this decreasing trend does not conform to the decreasing trend set for the second prediction condition—then it can be considered that the prediction of the target data mapped to the target segment as either hot or cold data has failed. In this case, no adjustment processing is required for the target segment, i.e., the step of executing the adjustment strategy for the target segment is not triggered.

[0068] In this implementation, the timing for implementing the adjustment strategy on the target shard is determined by setting predetermined adjustment trigger conditions. Specifically, when the trend of the access volume of the target shard within a predetermined period matches the trend of the target condition, the step of implementing the adjustment strategy on the target shard is triggered, thereby reducing the possibility of resource waste caused by unnecessary shard adjustments.

[0069] Optionally, in one implementation, step S101, which involves determining each piece of target data from the data stored in the target storage system, may include the following step B1:

[0070] Step B1: Based on the historical query volume per second of each data stored in the target storage system, generate a first sequence feature for each data that represents the access volume of each historical period; based on the first sequence feature of each data, determine the data in each data that meets the predetermined data prediction conditions to obtain each target data.

[0071] In this implementation, each piece of data stored in the target storage system corresponds to a historical query volume per second, which can be understood as the number of times the data is accessed per second in each historical period. Therefore, when determining each piece of target data from the data stored in the target storage system, a first sequence feature representing the access volume in each historical period can be generated for each piece of data based on the historical query volume per second. Then, based on the first sequence feature of each piece of data, data that meets the predetermined data prediction conditions is determined to obtain each piece of target data.

[0072] It should be noted that each of the above target data can be understood as: data that has a high probability of being accessed frequently or not frequently within a specified time period. For example, data a is frequently accessed between 14:54:59 and 15:01:59 every day, meaning data a is considered hot data during this period. Therefore, when performing hot data prediction at 14:49:59, data a can be used as a target data. As another example, data b is unlikely to be accessed between 23:50:59 and 23:59:59 every day, meaning data b is considered cold data during this period. Therefore, when performing cold data prediction at 23:45:59, data b can be used as a target data.

[0073] It should be emphasized that the purpose of this application embodiment is to alleviate the uneven load on the storage system in a timely manner by predicting the target data for future time periods. Therefore, the start time and end time corresponding to the above-mentioned predetermined duration are both within the time period corresponding to the above-mentioned specified duration.

[0074] Alternatively, in another implementation, step S101 above, determining each piece of target data from the data stored in the target storage system, may include the following step B2:

[0075] Step B2: Based on the client access information corresponding to each data stored in the target storage system, generate a second sequence feature for each data that represents the number of target clients in each historical period; based on the second sequence feature of each data, determine the data that meets the predetermined data prediction conditions in each data to obtain each target data; wherein, the target client is the client in the specified geographical area.

[0076] In this implementation, each piece of data stored in the target storage system corresponds to client access information. This client access information can be understood as the cumulative number of clients accessing the data in each historical period. The more clients accessing the data cumulatively, the greater the probability that the data is hot data; conversely, the fewer clients accessing the data cumulatively, the greater the probability that the data is cold data. Furthermore, considering the regional characteristics of the data—for example, during Singles' Day, clients in areas with free shipping access to online shopping apps far outnumber those in areas without free shipping—when performing data prediction, considering the regional characteristics of the data, a second sequence feature can be generated for each piece of data based on the client access information corresponding to each piece of data stored in the target storage system. This second sequence feature represents the number of target clients belonging to a specified geographical area in each historical period. Based on the second sequence feature of each piece of data, data that meets the predetermined data prediction conditions is determined to obtain each piece of target data.

[0077] It should be noted that each of the above target data can be understood as: data in each data set that indicates a higher probability of frequent access by target clients belonging to a specified geographical area within a specified time period, or a higher probability of infrequent access by target clients belonging to a specified geographical area.

[0078] In this implementation, the historical query volume per second corresponding to each piece of data stored in the target storage system, or the client access information corresponding to each piece of data, is used as the prediction benchmark for each piece of target data in the data stored in the target storage system, so as to improve the objectivity and reliability of the predicted hot or cold data.

[0079] Optionally, in one implementation, the step A above, which involves monitoring whether the target fragment meets the adjustment triggering condition, may include the following step A1:

[0080] Step A1: Monitor the target access data for the target fragment, and based on the obtained target access data, determine whether the target fragment meets the adjustment triggering conditions;

[0081] The target access data includes: the number of queries per second for the target shard at a predetermined time, and / or the difference between the number of first clients corresponding to the target shard and the number of second clients for the target shard; the predetermined time is the end time of a time period of predetermined duration; the number of first clients is: the number of clients that send access requests for the data mapped to the target shard within the predetermined duration; and the number of second clients is: the number of clients that send access requests for the data mapped to the target shard within any time period before the predetermined duration.

[0082] In this implementation, the number of queries per second for the target shard at a predetermined time, and / or the difference between the number of first clients and the number of second clients corresponding to the target shard, are used as the target access data for monitoring whether the target shard meets the adjustment triggering conditions.

[0083] When the target access data includes the number of queries per second for the target shard at a predetermined time, the historical number of queries per second for the target shard is obtained. Based on the obtained historical number of queries per second, a threshold for the number of queries per second is determined. This threshold is used as a criterion for judging whether the target shard meets the adjustment trigger condition. Thus, by comparing the monitored number of queries per second for the target shard at the predetermined time with the threshold, it is determined whether the target shard meets the adjustment trigger condition.

[0084] It should be noted that the threshold for queries per second mentioned above can be the historical average of queries per second, or a threshold corresponding to a specified percentage of the historical average of queries per second, etc., both of which are reasonable.

[0085] When the target access data includes the difference between the number of first clients corresponding to the target shard and the number of second clients corresponding to the target shard, the number of clients that made access requests for the data mapped to the target shard within a predetermined time period is obtained as the number of first clients corresponding to the target shard. The number of clients that made access requests for the data mapped to the target shard within any time period before the predetermined time period is obtained as the number of second clients corresponding to the target shard. Then, the difference between the number of first clients and the number of second clients is calculated. By comparing the calculated difference with a predetermined difference in the change of the number of clients, it is determined whether the target shard meets the adjustment trigger condition.

[0086] When the target access data includes the number of queries per second for the target shard at a predetermined time, and the difference between the number of first clients and the number of second clients corresponding to the target shard, it can be first determined whether the target shard meets the adjustment trigger condition by comparing the monitored number of queries per second for the target shard at the predetermined time with a threshold for the number of queries per second. If the comparison result indicates that the target shard does not meet the adjustment trigger condition, then the relationship between the calculated difference and a predetermined difference for the change in the number of clients is compared to reconfirm whether the target shard meets the adjustment trigger condition. In this way, by analyzing whether the target shard meets the adjustment trigger condition from both the number of queries per second and the number of clients, the accuracy of the analysis results regarding whether the target shard meets the adjustment trigger condition is improved.

[0087] It should be noted that when the target access data includes the number of queries per second (MS / s) of the target shard at a predetermined time, and the difference between the number of first clients and the number of second clients corresponding to the target shard, the execution order of analyzing the monitored relationship between the number of queries per second of the target shard at the predetermined time and a threshold for the number of queries per second, and analyzing the relationship between the calculated difference and a predetermined difference for the change in the number of clients, is as follows: First, analyze the monitored relationship between the number of queries per second of the target shard at the predetermined time and a threshold for the number of queries per second, and then analyze the relationship between the calculated difference and a predetermined difference for the change in the number of clients; alternatively, first analyze the relationship between the calculated difference and a predetermined difference for the change in the number of clients, and then analyze the monitored relationship between the number of queries per second of the target shard at the predetermined time and a threshold for the number of queries per second. Both are reasonable, and this application does not specifically limit the above execution order.

[0088] This implementation provides monitoring of whether the target shard meets the adjustment trigger conditions based on the number of queries per second and / or the number of clients. This enables accurate monitoring of the trigger timing of the target shard, reducing the inability to promptly alleviate uneven load in the storage system due to untimely shard adjustments.

[0089] Optionally, in one implementation, when the data prediction conditions include a first prediction condition, step A1 above, determining whether the target shard meets the adjustment trigger condition based on the obtained target access data, may include the following step A11:

[0090] Step A11: Determine whether the number of queries per second for the target shard at the predetermined time is higher than the first threshold determined based on the historical number of queries per second for the target shard. If so, determine that the target shard meets the adjustment trigger condition.

[0091] In this implementation, the historical query volume per second of the target shard is obtained, and a first threshold is determined based on the obtained historical query volume per second. This first threshold is then used as a criterion for determining whether the target shard meets the adjustment trigger condition, assuming the data prediction conditions include the first prediction condition. Thus, by monitoring whether the query volume per second of the target shard at a predetermined time exceeds the aforementioned first threshold, it is determined whether the target shard meets the adjustment trigger condition. If so, it indicates that the access volume of the target shard within a predetermined time period conforms to the changing trend set by the first prediction condition for at least one piece of target data; that is, the changing trend of the target shard's access volume is an increasing trend, and this increasing trend satisfies the increasing trend set by the first prediction condition. Therefore, the target shard meets the adjustment trigger condition.

[0092] It should be noted that the first threshold mentioned above can be the historical average number of queries per second, or a threshold corresponding to a specified proportion of the historical average number of queries per second, etc., both of which are reasonable.

[0093] Optionally, in one implementation, when the data prediction conditions include a first prediction condition, step A1 above, determining whether the target shard meets the adjustment trigger condition based on the obtained target access data, may include the following step A12:

[0094] Step A12: Determine whether the difference between the number of the first client corresponding to the target shard and the number of the second client corresponding to the target shard is greater than the first difference. If so, determine that the target shard meets the adjustment trigger condition.

[0095] In this implementation, by comparing the difference between the number of first clients corresponding to the target shard and the number of second clients corresponding to the target shard, and whether it is greater than the first difference, it is analyzed whether there is a surge in the number of accessing clients for the target shard within a predetermined time period. If so, it indicates that the access volume of the target shard within the predetermined time period conforms to the changing trend set by the first prediction condition for at least one target data, that is, the changing trend of the access volume of the target shard is an increasing trend, and the increasing trend satisfies the increasing trend set by the first prediction condition, and the target shard meets the adjustment trigger condition.

[0096] In this implementation, when the data prediction conditions include the first prediction condition, the target shard is monitored for whether it meets the adjustment trigger condition from two aspects: the number of queries per second at a predetermined time and whether there is a surge in the number of accessing clients within a predetermined time period. This enables accurate monitoring of the triggering timing of the target shard, thereby reducing the possibility that access requests cannot be distributed in a timely manner due to untimely sharding, and even increasing the risk of avalanche in the distributed storage system.

[0097] Optionally, in one implementation, when the data prediction conditions include a second prediction condition, step A1 above, determining whether the target shard meets the adjustment trigger condition based on the obtained target access data, may include the following step A13:

[0098] Step A13: Determine whether the number of queries per second for the target shard at the predetermined time is lower than the second threshold determined based on the historical number of queries per second for the target shard. If so, determine that the target shard meets the adjustment trigger condition.

[0099] In this implementation, the historical query volume per second of the target shard is obtained, and a second threshold is determined based on the obtained historical query volume per second. This second threshold is then used as a criterion for determining whether the target shard meets the adjustment trigger condition, assuming the data prediction conditions include the second prediction condition. Thus, by monitoring whether the query volume per second of the target shard at a predetermined time is lower than the aforementioned second threshold, it is determined whether the target shard meets the adjustment trigger condition. If so, it indicates that the access volume of the target shard within the predetermined time period conforms to the trend set by the second prediction condition for at least one piece of target data; that is, the trend of the access volume of the target shard is a downward trend, and this downward trend satisfies the downward trend set by the second prediction condition. Therefore, the target shard meets the adjustment trigger condition.

[0100] It should be noted that the second threshold mentioned above can be the historical average number of queries per second, or a threshold corresponding to a specified proportion of the historical average number of queries per second, etc., both of which are reasonable.

[0101] Optionally, in one implementation, when the data prediction condition includes the second prediction condition, step A1 above, determining whether the target shard meets the adjustment trigger condition based on the obtained target access data, may include the following step A14:

[0102] Step A14: Determine whether the difference between the number of first clients corresponding to the target shard and the number of second clients corresponding to the target shard is greater than the second difference. If so, determine that the target shard meets the adjustment trigger condition.

[0103] In this implementation, by comparing the difference between the number of first clients corresponding to the target shard and the number of second clients corresponding to the target shard, and whether it is greater than the second difference, it is analyzed whether there is a sharp decrease in the number of accessing clients for the target shard within a predetermined time period. If so, it indicates that the access volume of the target shard within the predetermined time period conforms to the changing trend set by the second prediction condition for at least one target data, that is, the changing trend of the access volume of the target shard is a downward trend, and the downward trend satisfies the downward trend set by the second prediction condition, and the target shard meets the adjustment trigger condition.

[0104] In this implementation, when the data prediction condition includes the second prediction condition, the target shard is monitored for whether it meets the adjustment trigger condition from two aspects: the number of queries per second at a predetermined time and whether there is a sharp decrease in the number of accessing clients within a predetermined time period. This enables accurate monitoring of the triggering timing of the target shard, thereby reducing the probability of an excessive number of idle shards due to untimely shard merging, which would prevent timely relief of the uneven load on the storage system.

[0105] Optionally, in one implementation, when the data prediction conditions include the first prediction condition, step S103 above, obtaining the adjustment strategy for the target shard, may include the following step C1:

[0106] Step C1: Analyze the required splitting method for the target shard according to the predetermined first analysis method, and determine the target node corresponding to each split shard obtained by splitting according to the required splitting method for the target shard. Use the required splitting method for the target shard and the target node corresponding to each split shard obtained by splitting as the adjustment strategy for the target shard.

[0107] Among them, such as Figure 2 As shown, the predetermined first analysis method may include the following steps S201-S204:

[0108] S201: Designate the target fragment as the first specified fragment to be split;

[0109] S202: Predict whether the query volume per second corresponding to each split fragment is lower than the predetermined query volume when the first specified fragment is split according to the target splitting method; if the query volume per second corresponding to each split fragment is lower than the predetermined query volume, proceed to step S203; otherwise, proceed to step S204.

[0110] The target splitting method is to split the data into a preset number of split shards. The query volume per second for each split shard is determined based on the data that the split shard needs to be mapped.

[0111] S203: The method used to characterize the splitting of the currently existing first specified fragment according to the target splitting method shall be used as the splitting method required for the target fragment;

[0112] S204: Select each target shard whose predicted query volume per second is not less than the predetermined query volume as the first designated shard to be split.

[0113] In this implementation, when the data prediction conditions include the first prediction condition, it is aimed at the first type of event (encountering a sudden high-frequency access to hot data). At this time, the determined target data is the predicted hot data. Correspondingly, the obtained adjustment strategy can be understood as a splitting strategy used to split the target fragment that maps at least one target data.

[0114] Based on this, when the data prediction conditions include the first prediction condition, after determining that the target shard is obtained, the required splitting method of the target shard is analyzed according to the predetermined first analysis method, and the target node corresponding to each split shard obtained by splitting according to the required splitting method of the target shard is determined. The required splitting method of the target shard and the target node corresponding to each split shard obtained by splitting are used as the adjustment strategy for the target shard.

[0115] Specifically, such as Figure 2 As shown, in the aforementioned predetermined first analysis method, firstly, the target shard is designated as the first specified shard to be split. Then, during the initial sharding process, the query volume per second (MS / s) of each of the predetermined number of split shards obtained when the existing first specified shard (i.e., the target shard) is split according to the target splitting method is predicted. This is used to analyze whether each MS / s is lower than the predetermined query volume, and based on the analysis results, it is determined whether to adopt the target splitting method to split the existing first specified shard. The predicted MS / s for each split shard is determined based on the data to be mapped to that split shard. For example, the historical MS / s for each piece of data to be mapped to that split shard is determined, and the sum of the determined historical MS / s is used as the MS / s for the entire split shard.

[0116] For example, if the query volume per second corresponding to each of the split shards is lower than the predetermined query volume, it indicates that each split shard obtained after splitting the target shard according to the above target splitting method can handle the required access requests when encountering sudden hot data access, and will not cause the distributed storage system to suffer from avalanche risk. In this case, the method used to characterize splitting the target shard according to the target splitting method can be used as the required splitting method for the target shard.

[0117] If there is a target splitting shard with a query volume per second that is not lower than the predetermined query volume for each splitting shard, it indicates that the target splitting shard may not be able to handle the required access requests when encountering sudden hot data access, or even cause the distributed storage system to be at risk of avalanche. In this case, each target splitting shard with a predicted query volume per second that is not lower than the predetermined query volume will be designated as the first specified shard to be split.

[0118] Then, predict whether the query volume per second corresponding to each split fragment is lower than the predetermined query volume when the existing first specified fragment (i.e., the target fragment and the target split fragment) is split according to the target splitting method.

[0119] If the query volume per second corresponding to each of the resulting partitions is lower than the predetermined query volume, it indicates that each partition obtained after partitioning the target partition and the currently targeted partition according to the aforementioned target partitioning method can handle the required access requests when encountering sudden hot data access, without causing a cascading failure risk in the distributed storage system. In this case, the method used to represent partitioning the existing first specified partition (i.e., the target partition and the currently targeted partition) according to the target partitioning method can be used as the partitioning method required for the target partition.

[0120] If a new target partition has a query volume per second that is not lower than the predetermined query volume for each partition obtained from the splitting process, it indicates that the new target partition obtained from the splitting process may not be able to handle the required access requests when encountering sudden hot data access, or may even lead to the risk of a cascading failure in the distributed storage system. In this case, each target partition with a predicted query volume per second that is not lower than the predetermined query volume is taken as the first designated partition to be split, and the above steps S202-S204 are repeated until the query volume per second for each partition obtained from the splitting process is lower than the predetermined query volume, and the splitting process ends.

[0121] It should be noted that the value corresponding to the above-mentioned predetermined query volume can be a pre-set value, the historical average query volume per second, or a threshold corresponding to a specified proportion of the historical average query volume per second, etc., all of which are reasonable.

[0122] Alternatively, in one embodiment, such as Figure 3 As shown, before splitting each target with a predicted query volume per second not less than a predetermined query volume into a shard as the first designated shard to be split, step S204 above may further include the following steps S301-S302:

[0123] S301: For each target split shard, predict whether the query volume per second of each virtual node is lower than the predetermined query volume when allocating at least one virtual node to the target split shard according to the shard popularity of the target split shard; if the query volume per second of each virtual node is lower than the predetermined query volume, then execute step S302, otherwise execute step S204.

[0124] Wherein, the shard heat corresponding to the target split shard is the heat of the data mapped by the target split shard;

[0125] S302: The method used to characterize the splitting of the currently existing first specified shard according to the target splitting method and the method of allocating at least one corresponding virtual node to each target splitting shard shall be used as the splitting method required for the target shard.

[0126] In this embodiment, considering that performing multiple sharding processes on the target shard would introduce multi-level sharding relationships, which could lead to an exponential increase in the complexity of the distributed storage system and increase maintenance costs, etc., before executing step S204 above, which designates each target shard with a predicted query volume per second not less than a predetermined query volume as the first designated shard to be sharded, virtual nodes are allocated to the shards to reduce the likelihood of the above problems and achieve overall load balancing for the shards. Specifically:

[0127] If any of the resulting shards has a query volume per second that is not lower than a predetermined query volume, it indicates that the resulting shard may be unable to handle the required access requests when encountering sudden surges in hot data access, potentially leading to a cascading failure risk in the distributed storage system. In this case, for each shard with a predicted query volume per second not lower than the predetermined query volume, the popularity of the data mapped to that shard is predicted as the shard popularity. This prediction then determines whether the query volume per second of each virtual node will be lower than the predetermined query volume when at least one virtual node is allocated to the shard according to its shard popularity.

[0128] If the query volume per second of each virtual node is lower than the predetermined query volume, it indicates that the virtual node is capable of handling the required access requests when encountering sudden hot data access. That is, the target partition to which the virtual node belongs can handle the required access requests as a whole. At this time, the method of splitting the existing first specified partition according to the target partitioning method and the method of allocating at least one corresponding virtual node to each target partitioning partition will be used as the partitioning method required by the target partitioning.

[0129] Otherwise, perform step S204 above, and use each target split shard whose predicted query volume per second is not less than the predetermined query volume as the first designated shard to be split.

[0130] It should be noted that when performing step S204 above, the virtual nodes allocated to the target split shards through S301 above will not be carried over into the new loop process. That is, when allocating virtual nodes to the target split shards again, it is not necessary to refer to the virtual nodes allocated to the target split shards in the previous loop process.

[0131] Optionally, for each shard's popularity, the number of virtual nodes allocated to that shard's popularity can be dynamically set. In this way, when allocating at least one virtual node to a target shard based on its popularity, the number of virtual nodes allocated to that shard's popularity can be determined based on its popularity, which serves as the number of nodes that should be allocated to that target shard. This ensures that shards with high popularity can occupy more ring positions in the entire hash ring, thereby reducing the possibility of untimely and scattered access requests during sudden hot data access.

[0132] In this embodiment, by allocating virtual nodes to the target shards, the complexity of the distributed storage system is reduced exponentially, which could lead to increased maintenance costs. At the same time, the shards can achieve load balancing. Thus, when encountering sudden hot data access, the target shards and the virtual nodes allocated to them can distribute access requests in a timely manner, reducing the possibility of a cascading failure in the distributed storage system.

[0133] Optionally, in one embodiment, before obtaining the adjustment strategy for the target fragment in step S103 above, the data processing method provided in this application embodiment may further include the following step D:

[0134] Step D: Determine whether the target fragment meets any of the predetermined fragment heat analysis conditions. If so, proceed with the step of obtaining the adjustment strategy for the target fragment.

[0135] The predetermined conditions for shard popularity analysis include: the current query volume per second of the target shard is higher than a predetermined value; or, the target shard is currently in a peak access period; or, the number of clients corresponding to the target shard is higher than a predetermined number; the peak access period is determined based on the time series characteristics of the historical access volume of the target shard.

[0136] In this embodiment, considering the distinction between cold and hot shards, a cold shard refers to a shard whose mapped data is typically not accessed frequently; in other words, the data mapped by a cold shard has low accessibility and is considered "cold data." Conversely, a hot shard refers to a shard whose mapped data is typically accessed frequently; in other words, the data mapped by a hot shard has high accessibility and is considered "hot data."

[0137] Therefore, when the target shard is a cold shard, if there is a sudden surge in high-frequency access to hot data (i.e., the data mapped to this shard is hot data), the node corresponding to this shard is likely to be able to handle the access requests for the hot data. In this case, there is no need to shard the target shard; otherwise, sharding is required. Based on this, shard popularity analysis conditions are pre-set to determine whether the target shard is a cold or hot shard, thereby determining whether to obtain an adjustment strategy for that target shard.

[0138] The conditions for shard popularity analysis include: when the current query volume per second of the target shard is higher than a predetermined value, it is determined whether the current query volume per second of the target shard is higher than the predetermined value. If it is higher, it indicates that the target shard is a hot shard. At this time, it is necessary to execute the step of obtaining the adjustment strategy for the target shard.

[0139] The conditions for shard popularity analysis include: when the target shard is currently in a peak access period, firstly, based on the time series characteristics of the historical access volume of the target shard, the peak access period of the target shard is analyzed to determine whether the target shard is currently in a peak access period. If it is, it indicates that the target shard is a hot shard. At this time, it is necessary to execute the step of obtaining the adjustment strategy for the target shard.

[0140] The conditions for shard popularity analysis include: when the number of clients corresponding to the target shard is higher than a predetermined number, the number of clients corresponding to the target shard is obtained. If the obtained number is higher than the predetermined number, it indicates that the target shard is a hot shard. At this time, it is necessary to execute the step of obtaining the adjustment strategy for the target shard.

[0141] It should be noted that the specific data settings for the above-mentioned predetermined values, peak access periods, and predetermined quantities are all for determining whether a shard is a hot shard. If you want to determine whether a target shard is a cold shard, the judgment process is the same as step D above, except that the specific data for the above-mentioned predetermined values, peak access periods, and predetermined quantities are different.

[0142] In this implementation, by setting predetermined fragmentation heat analysis conditions, it is possible to identify whether the target fragment is a hot fragment. When the target fragment is a hot fragment, the step of obtaining the adjustment strategy for the target fragment is executed so that the target fragment can be adjusted in a timely manner in the future.

[0143] Optionally, in one implementation, when the data prediction conditions include a second prediction condition, step S103 above, obtaining the adjustment strategy for the target shard, may include the following step C2:

[0144] Step C2: According to the predetermined second analysis method, analyze the merging method required for the target shard, and determine the target node corresponding to the merged shard obtained by merging according to the merging method required for the target shard. Use the merging method required for the target shard and the target node corresponding to the merged shard as the adjustment strategy for the target shard.

[0145] Among them, such as Figure 4 As shown, the predetermined second analysis method may include the following steps S401-S404:

[0146] S401: Designate the target fragment as the second specified fragment to be merged;

[0147] S402: Predict whether the query volume per second corresponding to the merged shard is lower than the predetermined query volume when merging the existing second specified shard according to the target merging method; if the query volume per second corresponding to the merged shard is lower than the predetermined query volume, then execute step S403; otherwise, execute step S404.

[0148] The target merging method is to merge the target shard with a preset number of other shards. The query volume per second corresponding to the merged shard is determined based on the data that needs to be mapped for the merged shard.

[0149] S403: The method used to characterize the currently existing second specified fragment for merging according to the target merging method shall be the merging method required by the target fragment;

[0150] S404: Select each target shard whose predicted query volume per second is not less than the predetermined query volume as the second designated shard to be merged.

[0151] In this implementation, when the data prediction condition includes the second prediction condition, it is aimed at the second type of event (the data popularity of a certain data changes from hot data to cold data). At this time, the determined target data is the predicted cold data. Correspondingly, the obtained adjustment strategy can be understood as a merging strategy used to merge target fragments that map at least one target data.

[0152] Based on this, when the data prediction conditions include the second prediction conditions, after determining the target shard, the required merging method of the target shard is analyzed according to the predetermined second analysis method, and the target node corresponding to the merged shard obtained by merging according to the required merging method of the target shard is determined. The required merging method of the target shard and the target node corresponding to the merged shard are used as the adjustment strategy for the target shard.

[0153] Specifically, such as Figure 4As shown, in the aforementioned predetermined second analysis method, firstly, the target shard is designated as the second specified shard to be merged. Then, during the initial shard merging, it is predicted whether the query volume per second (DPS) of the merged shard obtained by merging the existing second specified shard (i.e., the target shard) according to the target merging method is lower than a predetermined query volume. That is, when merging the target shard with a preset number of other shards, it is determined whether the DPS of the merged shard obtained is lower than the predetermined query volume based on the data that needs to be mapped for each merged shard. The predicted DPS for each merged shard is determined based on the data that needs to be mapped for that merged shard. For example, the sum of the historical DPS of each piece of data that needs to be mapped for that merged shard is used as the overall DPS for that merged shard.

[0154] If the query volume per second corresponding to the merged shards is lower than the predetermined query volume, it indicates that each merged shard obtained after merging the target shards according to the above target merging method can handle the required access requests when encountering situations where hot data is converted into cold data based on data popularity, without causing the risk of uneven load on the distributed storage system. In this case, the method used to characterize merging the target shards according to the target merging method can be taken as the required merging method for the target shards.

[0155] If there is a target merged shard with a query volume per second that is not lower than the predetermined query volume, it indicates that the target merged shard may be an idle shard when encountering a situation where hot data is converted into cold data according to data popularity. That is, although the shard can handle the required access requests, the existence of the shard may lead to a waste of storage system resources. Since the resources of the storage system are fixed, in order to further reduce the waste of resources, each target merged shard with a predicted query volume per second that is not lower than the predetermined query volume can be designated as the second specified shard to be merged.

[0156] Then, predict whether the query volume per second corresponding to each merged shard is lower than the predetermined query volume when the existing second specified shard (i.e., the target shard and the target merged shard) is merged according to the target merge method.

[0157] If the query volume per second for each merged shard is lower than the predetermined query volume, it indicates that each merged shard obtained by merging the target shard and the currently targeted merged shard according to the above target merging method can handle the required access requests when hot data is converted to cold data based on data popularity, without causing the risk of uneven load distribution in the distributed storage system. In this case, the method used to represent merging the existing second specified shard (i.e., the target shard and the currently targeted merged shard) according to the target merging method can be used as the merging method required for the target shard.

[0158] If a new target merged shard has a query volume per second that is not lower than the predetermined query volume for each merged shard, it indicates that the new target merged shard obtained in this merge may be an idle shard when encountering a situation where hot data is converted into cold data according to data popularity. That is, although the shard can handle the required access requests, the existence of the shard may lead to a waste of storage system resources. In this case, each target merged shard with a predicted query volume per second that is not lower than the predetermined query volume is designated as the second specified shard to be merged, and the above steps S402-S404 are repeated until the query volume per second for each merged shard obtained is lower than the predetermined query volume, and the merging ends.

[0159] It should be noted that the value corresponding to the above-mentioned predetermined query volume can be a pre-set value, the historical average query volume per second, or a threshold corresponding to a specified proportion of the historical average query volume per second, etc., all of which are reasonable.

[0160] Alternatively, in one embodiment, such as Figure 5 As shown, before step S404 above, which selects each target merge shard whose predicted query volume per second is not higher than the predetermined query volume as the second designated shard to be merged, may further include the following steps S501-S502:

[0161] S501: Predict whether the query volume per second of each remaining virtual node is lower than the predetermined query volume after the virtual nodes are reduced according to the predetermined node reduction method; if the query volume per second of each remaining virtual node is lower than the predetermined query volume, then execute step S502; otherwise, execute step S404.

[0162] The predetermined node reduction method is based on the shard popularity corresponding to the target merge shard, which is at least one virtual node corresponding to the reduction of the target merge shard; the shard popularity corresponding to the target merge shard is the popularity of the data mapped by the target merge shard;

[0163] S502: The method used to characterize the merging of the existing second specified shard according to the target merging method and the method of reducing at least one virtual node corresponding to the target merging shard are taken as the merging method required for the target shard.

[0164] In this embodiment, the virtual nodes configured in the shards also occupy a certain amount of storage system resources. Therefore, before executing step S404 above, which selects each target merge shard with a predicted query volume per second not less than a predetermined query volume as the second designated shard to be merged, the resource waste caused by idle virtual nodes is reduced by decreasing the number of virtual nodes corresponding to the merged shards. Specifically:

[0165] If the merged shards have a query volume per second that is not lower than a predetermined query volume, it indicates that the merged shard may be an idle shard when hot data is converted to cold data based on data popularity. The virtual nodes of the corresponding shard are also idle virtual nodes. That is, although the shard and the corresponding virtual nodes can handle the required access requests, their existence may lead to a waste of storage system resources. In this case, for each target merged shard with a predicted query volume per second that is not lower than the predetermined query volume, the popularity of the data mapped by the target merged shard is predicted as the shard popularity corresponding to the target merged shard. Thus, it is predicted whether the query volume per second of each remaining virtual node is higher than the predetermined query rate when at least one virtual node is reduced for the target merged shard according to the shard popularity corresponding to the target merged shard.

[0166] If the query volume per second of each remaining virtual node is lower than the predetermined query volume, it indicates that the virtual node can handle the required access requests when encountering situations where hot data is converted into cold data according to data popularity. That is, the target merge shard to which the virtual node belongs can handle the required access situation. At this time, it will be used to characterize the method of merging the existing second specified shard according to the target merging method, and the method of reducing at least one virtual node corresponding to the target merge shard, as the merging method required for the target shard.

[0167] Otherwise, perform step S404 above, and use each target merge shard whose predicted query volume per second is not less than the predetermined query volume as the second designated shard to be merged.

[0168] It should be noted that when performing step S404 above, the virtual nodes reduced for the target sharding in step S501 above will not be carried over into the new loop process. That is, when reducing virtual nodes for the target sharding again, it is not necessary to refer to the virtual nodes reduced for the target sharding in the previous loop process.

[0169] Optionally, for each shard's heat level, the number of virtual nodes to be reduced corresponding to that shard's heat level can be dynamically set. In this way, when reducing at least one virtual node corresponding to the shard's heat level for the target shard merge, the number of virtual nodes to be reduced corresponding to that shard's heat level can be determined as the number of nodes to be reduced for the target shard merge. This ensures that shards with low heat level occupy fewer ring positions in the entire hash ring, thereby reducing resource waste in the storage system.

[0170] In this embodiment, by reducing the number of virtual nodes for the target shard merging, the overall load balancing effect of the merged shards can be achieved, while further reducing the waste of storage system resources caused by idle shards.

[0171] Optionally, when the data prediction condition is the second prediction condition, after the adjustment strategy is executed on the target shard, the adjustment strategy is retained. After a certain period of time after the adjustment strategy is executed, if it is detected that the data mapped by the target shard is converted from cold data back to hot data in a short time, the merged shard is restored to the configuration before the adjustment strategy was executed based on the adjustment strategy for the target shard. Otherwise, after the aforementioned certain period of time, the adjustment strategy is deleted.

[0172] To facilitate understanding, a specific example will be used below to describe in detail a data processing method provided in the embodiments of this application.

[0173] With the development of AI applications such as model training, the amount and quantity of training data are increasing, and much of this data access is based on the POSIX interface (Portable Operating System Interface of UNIX). In a cloud-native environment, current distributed file storage systems face two core challenges:

[0174] One issue is the avalanche problem of metadata services caused by sudden hot access. This is because text containing hot data is accessed frequently, leading to uneven load on the corresponding storage nodes, resulting in performance degradation and potentially causing a valanche risk in the metadata service. To address this issue, traditional solutions use static hash ring sharding, such as Ceph CRUSH (Ceph Controlled Replication Under Scalable Hashing), to further shard the shards on the hash ring. However, these traditional solutions have two major drawbacks: (1) shard splitting requires manual intervention and is extremely time-consuming; (2) the replica rebalancing process may cause increased request latency.

[0175] Secondly, in dynamic scaling scenarios, existing systems such as HDFS Federation (Hadoop Distributed File System Federation) adopt a pre-allocated partition mode. Scaling up requires downtime to migrate data, which leads to a degradation in the service performance of the distributed storage system and low efficiency in dynamic scaling. This is because traditional hash rings require global data migration when nodes are added or removed, which is slow and prone to resource waste.

[0176] Based on this, this specific embodiment achieves efficient load balancing and rapid scaling by optimizing intelligent elastic indexes, combined with dynamic sharding, predictive models, and containerized orchestration technology.

[0177] This specific embodiment combines the temporal prediction capability of neural networks with distributed system control theory. It analyzes the historical access sequences of data throughout the distributed storage system using an LSTM (Long Short-Term Memory) model to obtain a hotspot distribution probability matrix for the next 5 minutes, starting from the current time. Based on this matrix, it predicts whether the data is hot or cold, designating it as target data. Then, based on each target data obtained, it determines a target shard for mapping at least one target data. For each target shard, it drives a hot weight allocation splitting decision engine (i.e., the adjustment strategy corresponding to the target shard in this application). When the predicted value of a target shard exceeds a dynamic threshold (i.e., when the target shard meets a predetermined adjustment trigger condition), it triggers a containerized fast splitting or merging process. The formula for calculating the dynamic threshold is Threshold = Current Load + Predicted Increment. Threshold represents the trend of the access volume to the target shard within a predetermined period of time, satisfying the trend of the trend set for the target conditions. Current Load represents the access volume to the target shard at the current moment. Predicted Increment represents the predicted access volume to the target shard between the current moment and the end of the predetermined period.

[0178] For example, the above data can be feature values ​​of multiple features such as queries per second, read / write times per second, data affinity index, network throughput, access time limitation, periodic access intensity, regional access density, processor load balancing, memory cache hit rate, request type weight, tenant priority, index level depth, etc. This application does not make specific limitations on these.

[0179] The following is combined Figure 6 This specific embodiment will be described below, such as Figure 6 As shown, in this specific embodiment, the hash ring set in the distributed storage system is a multi-layer hash ring design. Specifically, the distributed storage system sets up a region-level global index hash ring (i.e., the global hash ring in this application) and three node-level region hash rings. Each region hash ring is divided into two shards by a node. It should be noted that the data range that each node in the global index hash ring can map is larger than the data range that each node in the region hash ring can map, and the data range that each node in the region hash ring can map is larger than the data range that each shard can map. Thus, when determining the target shard for each target data, firstly, the hash value is mapped to the global index hash ring to determine the region hash ring whose mapped range contains the hash value. Then, from the region hash rings, the shards whose mapped range contains the hash value are determined to obtain the target shard used to map the target data.

[0180] based on Figure 6 The multi-layer hash ring design shown in this specific embodiment can map each target data to its corresponding target node after the target shard is sharded. For regional hash rings and global index hash rings, this process can be understood as triggering the cluster node to expand (e.g., add metadata worker nodes) or shrink (e.g. delete metadata worker nodes) through the QoS (Quality of Service) aware controller.

[0181] The following is combined Figure 7 The schematic diagram shown illustrates the implementation process of this specific embodiment:

[0182] Relevant monitoring data is collected from each metadata working node within the file system index, and then the collected monitoring data is delivered to the long short-term memory network model in the spatiotemporal joint prediction model through the monitoring service.

[0183] The Long Short-Term Memory (LSTM) network model in the spatiotemporal joint prediction model receives data for inference and prediction. From a temporal perspective, it analyzes the historical query volume per second for each data point in the distributed storage system, predicting future access trends for each data point as the prediction result (i.e., step B1 in this application, where the obtained prediction result is the target data in this application). Alternatively, from a spatial perspective, it combines the client's IP geographic location to predict future regional access hotspot data as the prediction result (i.e., step B2 in this application). Then, based on the obtained prediction results, a target shard for mapping the target data is determined. This allows for the pre-generation of a contingency plan for the target shard (i.e., the step in this application to obtain the adjustment strategy for the target shard), to pre-expand potential hot shards and reduce shard splitting latency from minutes to seconds. For example, predicting a surge in access to a product page during an e-commerce promotion and pre-splitting the corresponding shard; or, pre-shrinking potential cold shards and reducing shard merging latency from minutes to seconds. For example, predicting a decrease in access to TV series L after its conclusion and pre-merging the corresponding shard.

[0184] The analysis process for potential hot / cold shards (also known as the execution process of the dynamic topology-aware sharding algorithm, i.e., the process of determining whether a target shard meets any of the predetermined sharding heat analysis conditions in this application; conversely, the process of determining whether a target shard is a cold shard is similar to the process of determining whether a target shard is a hot shard, only the judgment benchmark values ​​are different, and will not be elaborated here) specifically involves: determining whether a shard is a potential hot / cold shard based on access frequency, i.e., counting the number of queries per second of the shard; or determining whether a shard is a potential hot / cold shard based on temporal characteristics, i.e., identifying access patterns and determining whether the shard is in a periodic access peak period; or determining whether a shard is a potential hot / cold shard based on client distribution, i.e., clustering access sources according to geographical location or user groups.

[0185] For each target fragment belonging to hot fragmentation, a corresponding splitting strategy is generated. The execution process of this splitting strategy can be understood as a pre-fragmentation splitting process performed on the target fragment. This process is as follows:

[0186] First, the target shard is horizontally expanded, that is, it is split into multiple sub-shards (for example, the original shard Q is split into sub-shards Q1 and Q2). If the query volume per second of each sub-shard (i.e., the split shard in this application) is less than 50% of the dynamic threshold (i.e., the first threshold in this application), it indicates that the split sub-shards can handle the burst access of hot data. In this case, the splitting method obtained after the pre-sharding is used as the splitting strategy for the target shard. If the query volume per second of one sub-shard after splitting is not less than 50%, it indicates that the split sub-shards can handle the burst access of hot data. When the dynamic threshold reaches 50%, the shard heat corresponding to each sub-shard obtained by splitting needs to be calculated. Then, according to the weight corresponding to the shard heat of the sub-shard, virtual nodes are allocated to the sub-shard to ensure load balancing. If the query volume per second of each allocated virtual node is less than 50% of the dynamic threshold, it indicates that the load balancing of the entire hash ring can be guaranteed by the above splitting method and the allocation method of virtual nodes. At this time, the splitting method and allocation method obtained after the above pre-sharding are used as the splitting strategy for the target shard.

[0187] For target fragments that are considered cold fragments, a corresponding merging strategy is generated. The execution process of this merging strategy can be understood as a pre-fragmentation and merging process performed on the target fragment. This process is as follows:

[0188] First, the target shard is horizontally scaled down, that is, the target shard is merged with other shards into the same shard. If the query volume per second of each merged shard is less than 50% of the dynamic threshold mentioned above (i.e., the second threshold in this application), it indicates that the merged shard can handle the required access requests. In this case, the merging method obtained after the pre-merging is used as the merging strategy for the target shard. If the query volume per second of a shard after merging is not less than 50% of the dynamic threshold mentioned above, the shard heat corresponding to each merged sub-shard needs to be calculated. Then, according to the weight corresponding to the shard heat of the merged shard, the virtual nodes of the merged shard are reduced to ensure that the resources of the storage system are effectively utilized. If the query volume per second of each remaining virtual node after reduction is less than 50% of the dynamic threshold mentioned above, it indicates that the load balance of the entire hash ring can be guaranteed according to the merging method and the virtual node reduction method mentioned above. In this case, the merging method and reduction method obtained after the pre-merging are used as the merging strategy for the target shard.

[0189] It should be noted that in this specific embodiment, a triggering condition is set for triggering the execution of the shard splitting process (i.e., the splitting strategy). The triggering condition may include: triggering the execution of the shard splitting process (i.e., step A11 in this application) when the query volume per second of the target shard is higher than the dynamic threshold calculated based on the historical load (historical query volume) of the target shard; or triggering the execution of the shard splitting process (i.e., step A12 in this application) when the concentration of client distribution accessing the data mapped by the target shard suddenly increases, for example, when the access request volume of a certain region surges.

[0190] Therefore, when the target shard meets the above triggering conditions, the corresponding shard splitting process (i.e., the corresponding splitting strategy) is executed, and sub-shard Pods (container groups, i.e., the target nodes in this application) are created in real time via the Kubernetes API (Kubernetes Application Programming Interface), injecting a new hash ring. The above shard splitting process can be understood as follows: the data to be mapped by the original shard is split into various split shards, and the data to be mapped by the original shard is mapped by each of the split shards. In other words, the data that the original shard needed to map before the split is divided into different split shards. The creation of the sub-shard Pods essentially involves dividing a shard into different regions by setting up nodes. The sum of the data to be mapped in each region is the data that the shard itself needs to map.

[0191] Furthermore, in this specific embodiment, a triggering condition is set for triggering the execution of the shard merging process (i.e., the merging strategy). The triggering condition may include: triggering the execution of the shard merging process (i.e., step A13 in this application) when the query volume per second of the target shard is lower than the dynamic threshold calculated based on the historical load (historical query volume) of the target shard; or triggering the execution of the shard merging process (i.e., step A14 in this application) when the concentration of client distribution accessing the data mapped by the target shard decreases, for example, when the access request volume of a certain region decreases.

[0192] Therefore, when the target shard meets the above triggering conditions, the corresponding shard merging process (i.e., the corresponding merging strategy) is executed, and the shard Pod (container group, i.e. the target node in this application) is reduced in real time through the Kubernetes API (Kubernetes Application Programming Interface), and the corresponding node is removed from the hash ring.

[0193] It should be noted that during the aforementioned shard splitting and shard merging processes, a shard migration process is involved. During this process, only the difference data needs to be synchronized; a full copy is not required. That is, only the data mapped to each sub-shard is migrated from the target shard to that sub-shard. Furthermore, if new data is added to the target shard during the shard migration process, only the original data mapped to the new data in the target shard needs to be migrated first, and then migrated to the corresponding sub-shard of the target shard. This utilizes incremental logging to reduce network transmission volume.

[0194] It is important to emphasize that when performing metadata synchronization, the Raft protocol is optimized to limit the metadata synchronization scope to the relevant shard group (i.e., the shard group corresponding to the target shard) in order to achieve low-latency metadata synchronization.

[0195] In this specific embodiment, the dynamic index splitting scheme based on LSTM prediction reduces the access latency of hot data, alleviates the access pressure of hot indexes, and reduces the performance impact of business on data access. Furthermore, this specific embodiment, through the LSTM prediction-driven contingency plan, can significantly reduce the cost of advance planning and human intervention in operation and maintenance by shortening the decision time for scaling up / down, without the need for excessive advance resource planning. For the business, this improves transparency, and due to the elastic access of dynamic indexes, it reduces the operation and maintenance costs of manually scaling up and down indexes.

[0196] Based on the above method embodiments, this application provides a data processing apparatus, such as... Figure 8 As shown, the device includes:

[0197] The first determining module 810 is used to determine each piece of target data from the data stored in the target storage system; wherein each piece of target data is data that meets predetermined data prediction conditions; the predetermined data prediction conditions include a first prediction condition for predicting hot data, or a second prediction condition for predicting cold data.

[0198] The second determining module 820 is used to determine the target fragment in the hash ring set by the target storage system based on the target data; wherein the target fragment is a fragment used to map at least one target data.

[0199] The acquisition module 830 is used to acquire the adjustment strategy for the target shard; wherein, the adjustment strategy is used to indicate the adjustment method of the target shard and the target node corresponding to the shard obtained after adjustment;

[0200] The execution module 840 is used to execute the adjustment strategy on the target shard to adjust the target shard according to the adjustment method, and to map the at least one target data to its corresponding target node; wherein, the target node corresponding to each target data is the target node corresponding to the shard to which the target data needs to be mapped.

[0201] Optionally, in one implementation, the apparatus further includes:

[0202] The monitoring module is used to monitor whether the target shard meets the adjustment triggering condition before the adjustment strategy is executed on the target shard; and to trigger the execution module 840 when the adjustment triggering condition is met.

[0203] The adjustment trigger condition is used to characterize the trend of changes in the access volume of the target segment within a predetermined time period, which matches the trend of changes set for the target condition; the target condition is the prediction condition included in the data prediction condition that is satisfied by the at least one target data.

[0204] Optionally, in one implementation, the monitoring module is specifically used for:

[0205] Based on the historical query volume per second of each data stored in the target storage system, a first sequence feature representing the access volume of each historical period is generated for each data; based on the first sequence feature of each data, the data that meets the predetermined data prediction conditions are determined to obtain each target data.

[0206] or,

[0207] Based on the client access information corresponding to each data stored in the target storage system, a second sequence feature is generated for each data, representing the number of target clients in each historical period; based on the second sequence feature of each data, data that meets the predetermined data prediction conditions are determined to obtain each target data; wherein, the target client is a client within a specified geographical area.

[0208] Optionally, in one implementation, the monitoring module is specifically used to include:

[0209] Monitor target access data for the target fragment, and based on the obtained target access data, determine whether the target fragment meets the adjustment triggering conditions;

[0210] The target access data includes: the number of queries per second for the target shard at a predetermined time, and / or the difference between the number of first clients corresponding to the target shard and the number of second clients for the target shard;

[0211] The predetermined time is the end time of the predetermined duration period; the first client number is the number of clients that issue access requests for the data mapped to the target shard within the predetermined duration period; the second client number is the number of clients that issue access requests for the data mapped to the target shard within any time period before the predetermined duration period.

[0212] Optionally, in one implementation, when the data prediction condition includes the first prediction condition, the monitoring module is specifically used for:

[0213] Determine whether the number of queries per second for the target shard at the predetermined time is higher than a first threshold determined based on the historical number of queries per second for the target shard. If so, determine that the target shard meets the adjustment trigger condition; and / or, determine whether the difference between the number of first clients corresponding to the target shard and the number of second clients corresponding to the target shard is greater than a first difference. If so, determine that the target shard meets the adjustment trigger condition.

[0214] Optionally, in one implementation, when the data prediction conditions include the second prediction condition, the monitoring module is specifically used for:

[0215] Determine whether the number of queries per second for the target shard at the predetermined time is lower than a second threshold determined based on the historical number of queries per second for the target shard. If so, determine that the target shard meets the adjustment trigger condition; and / or, determine whether the difference between the number of first clients corresponding to the target shard and the number of second clients corresponding to the target shard is greater than a second difference. If so, determine that the target shard meets the adjustment trigger condition.

[0216] Optionally, in one implementation, the acquisition module 830 is specifically used for:

[0217] When the data prediction conditions include the first prediction conditions, the splitting method required for the target shard is analyzed according to a predetermined first analysis method, and the target node corresponding to each split shard obtained by splitting according to the splitting method required for the target shard is determined. The splitting method required for the target shard and the target node corresponding to each split shard obtained by splitting are used as the adjustment strategy for the target shard.

[0218] The predetermined first analysis method includes:

[0219] The target fragment is designated as the first specified fragment to be split.

[0220] Predict whether the query volume per second corresponding to each split fragment is lower than a predetermined query volume when the existing first specified fragment is split according to the target splitting method; the target splitting method is the method of splitting to obtain a preset number of split fragments, and the query volume per second corresponding to each split fragment is determined based on the data to be mapped for that split fragment;

[0221] If the query volume per second corresponding to each split fragment is lower than the predetermined query volume, it will be used to characterize the way the existing first specified fragment is split according to the target splitting method, and will be used as the splitting method required for the target fragment;

[0222] Otherwise, each target shard whose predicted query volume per second is not less than the predetermined query volume will be designated as the first specified shard to be split.

[0223] Optionally, in one implementation, the apparatus further includes:

[0224] The node sharding module is used to, before designating each target shard with a predicted query volume per second not less than a predetermined query volume as the first designated shard to be sharded, predict whether the query volume per second of each virtual node is less than the predetermined query volume when allocating at least one corresponding virtual node to the target shard according to the shard heat corresponding to the target shard; wherein, the shard heat corresponding to the target shard is the heat of the data mapped by the target sub-shard;

[0225] If the query volume per second of each virtual node is lower than the predetermined query volume, it will be used to characterize the way to split the existing first specified shard according to the target splitting method and the way to allocate the corresponding at least one virtual node to each target splitting shard, as the splitting method required for the target shard.

[0226] Otherwise, the step of determining each target shard whose predicted query volume per second is not less than the predetermined query volume is triggered as the first designated shard to be split.

[0227] Optionally, in one implementation, the apparatus further includes:

[0228] The heat analysis module is used to determine whether the target segment meets any of the predetermined segment heat analysis conditions before obtaining the adjustment strategy for the target segment. If so, the step of obtaining the splitting strategy for the target segment is executed.

[0229] The predetermined segmented heat analysis conditions include:

[0230] The current query volume per second for the target shard is higher than the predetermined value;

[0231] Alternatively, the target shard is currently experiencing a peak access period; the peak access period is determined based on the time-series characteristics of the target shard's historical access volume.

[0232] Alternatively, the number of clients currently corresponding to the target shard is higher than a predetermined number.

[0233] Optionally, in one implementation, the acquisition module 830 is specifically used for:

[0234] When the data prediction conditions include the second prediction conditions, the required merging method for the target shard is analyzed according to a predetermined second analysis method, and the target node corresponding to the merged shard obtained by merging according to the required merging method for the target shard is determined. The required merging method for the target shard and the target node corresponding to the merged shard obtained are used as the adjustment strategy for the target shard.

[0235] The predetermined second analysis method includes:

[0236] The target fragment is designated as the second specified fragment to be merged.

[0237] Predict whether the query volume per second corresponding to the merged shard is lower than a predetermined query volume when merging the currently existing second specified shard according to the target merging method; the target merging method is to merge the target shard with a preset number of other shards, and the query volume per second corresponding to the merged shard is determined based on the data that the merged shard needs to be mapped;

[0238] If the query volume per second corresponding to the merged shard is lower than the predetermined query volume, it will be used to characterize the current existing second specified shard being merged according to the target merging method, and will be used as the merging method required for the target shard.

[0239] Otherwise, each target merge shard whose predicted query volume per second is not less than the predetermined query volume will be designated as the second specified shard to be merged.

[0240] Optionally, in one implementation, the apparatus further includes:

[0241] The node reduction module is used to predict, before designating each target shard whose predicted query volume per second is not higher than the predetermined query volume as the second designated shard to be merged, whether the query volume per second of each remaining virtual node is lower than the predetermined query volume after virtual node reduction according to a predetermined node reduction method; wherein, the predetermined node reduction method is based on the shard popularity corresponding to the target shard to be merged, and is at least one virtual node corresponding to the reduction of the target shard to be merged; the shard popularity corresponding to the target shard to be merged is the popularity of the data mapped by the target shard to be merged;

[0242] If the query volume per second of each remaining virtual node is lower than the predetermined query volume, it will be used to characterize the method of merging the currently existing second specified shard according to the target merging method and the method of reducing the at least one virtual node corresponding to the target merged shard, as the merging method required for the target shard;

[0243] Otherwise, the step of designating the merged fragment as the second specified fragment to be merged is triggered.

[0244] This application also provides an electronic device, such as... Figure 9 As shown, it includes a processor 901, a communication interface 902, a memory 903, and a communication bus 904, wherein the processor 901, the communication interface 902, and the memory 903 communicate with each other through the communication bus 904.

[0245] Memory 903 is used to store computer programs;

[0246] When the processor 901 executes the program stored in the memory 903, it implements any of the data processing methods described in the above embodiments.

[0247] The communication bus mentioned above can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not mean that there is only one bus or one type of bus.

[0248] The communication interface is used for communication between the aforementioned terminal and other devices.

[0249] The memory may include random access memory (RAM) or non-volatile memory, such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0250] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0251] In another embodiment provided in this application, a computer-readable storage medium is also provided, wherein a computer program is stored therein, and when the computer program is executed by a processor, it implements any of the data processing methods described in the above embodiments.

[0252] In another embodiment provided in this application, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to perform any of the data processing methods described in the above embodiments.

[0253] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid state disk (SSD)).

[0254] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0255] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the device embodiments, electronic device embodiments, computer-readable storage medium embodiments, and computer program product embodiments are basically similar to the method embodiments, so the descriptions are relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0256] The above description is merely a preferred embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application are included within the scope of protection of this application.

Claims

1. A data processing method, characterized in that, include: From the data stored in the target storage system, each target data is determined; wherein each target data is data that meets predetermined data prediction conditions; the predetermined data prediction conditions include a first prediction condition for predicting hot data, or a second prediction condition for predicting cold data. Based on the target data, the target fragment in the hash ring set by the target storage system is determined; wherein, the target fragment is a fragment used to map at least one target data. Obtain the adjustment strategy for the target shard; wherein the adjustment strategy is used to indicate the adjustment method of the target shard and the target node corresponding to the shard obtained after adjustment; The adjustment strategy is executed on the target shard to adjust the target shard according to the adjustment method, and the at least one target data is mapped to its corresponding target node; wherein, the target node corresponding to each target data is the target node corresponding to the shard to which the target data needs to be mapped.

2. The method according to claim 1, characterized in that, Before applying the adjustment strategy to the target fragment, the method further includes: Monitor whether the target fragment meets the adjustment trigger condition; When the adjustment triggering condition is met, the step of executing the adjustment strategy on the target fragment is triggered; The adjustment trigger condition is used to characterize the trend of changes in the access volume of the target segment within a predetermined time period, which matches the trend of changes set for the target condition; the target condition is the prediction condition included in the data prediction condition that is satisfied by the at least one target data.

3. The method according to claim 2, characterized in that, The process of determining each piece of target data from the data stored in the target storage system includes: Based on the historical query volume per second of each data stored in the target storage system, a first sequence feature representing the access volume of each historical period is generated for each data; based on the first sequence feature of each data, the data that meets the predetermined data prediction conditions are determined to obtain each target data. or, Based on the client access information corresponding to each data stored in the target storage system, a second sequence feature is generated for each data, representing the number of target clients in each historical period; based on the second sequence feature of each data, data that meets the predetermined data prediction conditions are determined to obtain each target data; wherein, the target client is a client within a specified geographical area.

4. The method according to claim 2, characterized in that, The monitoring of whether the target fragment meets the adjustment trigger condition includes: Monitor target access data for the target fragment, and based on the obtained target access data, determine whether the target fragment meets the adjustment triggering conditions; The target access data includes: the number of queries per second for the target shard at a predetermined time, and / or the difference between the number of first clients corresponding to the target shard and the number of second clients for the target shard; The predetermined time is the end time of the predetermined duration period; the first client number is the number of clients that issue access requests for the data mapped to the target shard within the predetermined duration period; the second client number is the number of clients that issue access requests for the data mapped to the target shard within any time period before the predetermined duration period.

5. The method according to claim 4, characterized in that, When the data prediction condition includes the first prediction condition, determining whether the target shard meets the adjustment trigger condition based on the obtained target access data includes: Determine whether the number of queries per second for the target shard at the predetermined time is higher than a first threshold determined based on the historical number of queries per second for the target shard. If so, determine that the target shard meets the adjustment trigger condition; and / or, determine whether the difference between the number of first clients corresponding to the target shard and the number of second clients corresponding to the target shard is greater than a first difference. If so, determine that the target shard meets the adjustment trigger condition.

6. The method according to claim 4, characterized in that, When the data prediction condition includes the second prediction condition, determining whether the target shard meets the adjustment trigger condition based on the obtained target access data includes: Determine whether the number of queries per second for the target shard at the predetermined time is lower than a second threshold determined based on the historical number of queries per second for the target shard. If so, determine whether the target shard meets the adjustment trigger condition; and / or, determine whether the difference between the number of first clients corresponding to the target shard and the number of second clients corresponding to the target shard is greater than a second difference. If so, determine whether the target shard meets the adjustment trigger condition.

7. The method according to claim 1 or 2, characterized in that, When the data prediction conditions include the first prediction condition, obtaining the adjustment strategy for the target shard includes: According to the predetermined first analysis method, the required splitting method of the target fragment is analyzed, and the target node corresponding to each split fragment obtained by splitting according to the required splitting method of the target fragment is determined. The required splitting method of the target fragment and the target node corresponding to each split fragment obtained by splitting are used as the adjustment strategy for the target fragment. The predetermined first analysis method includes: The target fragment is designated as the first specified fragment to be split. Predict whether the query volume per second corresponding to each split fragment is lower than a predetermined query volume when the existing first specified fragment is split according to the target splitting method; the target splitting method is the method of splitting to obtain a preset number of split fragments, and the query volume per second corresponding to each split fragment is determined based on the data to be mapped for that split fragment; If the query volume per second corresponding to each split fragment is lower than the predetermined query volume, it will be used to characterize the way the existing first specified fragment is split according to the target splitting method, and will be used as the splitting method required for the target fragment; Otherwise, each target shard whose predicted query volume per second is not less than the predetermined query volume will be designated as the first specified shard to be split.

8. The method according to claim 7, characterized in that, Before designating each target shard with a predicted query volume per second not less than a predetermined query volume as the first designated shard to be split, the method further includes: For each target split shard, predict whether the query volume per second of each virtual node is lower than the predetermined query volume when at least one virtual node is allocated to the target split shard according to the shard popularity of the target split shard; wherein, the shard popularity of the target split shard is the popularity of the data mapped by the target split shard. If the query volume per second of each virtual node is lower than the predetermined query volume, it will be used to characterize the way to split the existing first specified shard according to the target splitting method and the way to allocate the corresponding at least one virtual node to each target splitting shard, as the splitting method required for the target shard. Otherwise, the step of determining each target shard whose predicted query volume per second is not less than the predetermined query volume is triggered as the first designated shard to be split.

9. The method according to claim 1, characterized in that, Before obtaining the adjustment strategy for the target fragment, the method further includes: Determine whether the target fragment meets any of the predetermined fragment heat analysis conditions. If so, execute the step of obtaining the adjustment strategy for the target fragment. The predetermined segmented heat analysis conditions include: The current query volume per second for the target shard is higher than the predetermined value; Alternatively, the target shard is currently experiencing a peak access period; the peak access period is determined based on the time-series characteristics of the target shard's historical access volume. Alternatively, the number of clients currently corresponding to the target shard is higher than a predetermined number.

10. The method according to claim 1 or 2, characterized in that, When the data prediction conditions include the second prediction condition, obtaining the adjustment strategy for the target shard includes: According to the predetermined second analysis method, the required merging method of the target shard is analyzed, and the target node corresponding to the merged shard obtained by merging according to the required merging method of the target shard is determined. The required merging method of the target shard and the target node corresponding to the merged shard are used as the adjustment strategy of the target shard. The predetermined second analysis method includes: The target fragment is designated as the second specified fragment to be merged. Predict whether the query volume per second corresponding to the merged shard is lower than a predetermined query volume when merging the currently existing second specified shard according to the target merging method; the target merging method is to merge the target shard with a preset number of other shards, and the query volume per second corresponding to the merged shard is determined based on the data that the merged shard needs to be mapped; If the query volume per second corresponding to the merged shard is lower than the predetermined query volume, it will be used to characterize the current existing second specified shard being merged according to the target merging method, and will be used as the merging method required for the target shard. Otherwise, each target merge shard whose predicted query volume per second is not less than the predetermined query volume will be designated as the second specified shard to be merged.

11. The method according to claim 10, characterized in that, Before designating each target merge shard whose predicted query volume per second is not higher than the predetermined query volume as the second designated shard to be merged, the method further includes: Predict whether the query volume per second of each remaining virtual node is lower than the predetermined query volume after virtual nodes are reduced according to a predetermined node reduction method; wherein, the predetermined node reduction method is based on the shard popularity corresponding to the target merge shard, and is at least one virtual node corresponding to the reduction of the target merge shard; the shard popularity corresponding to the target merge shard is the popularity of the data mapped by the target merge shard; If the query volume per second of each remaining virtual node is lower than the predetermined query volume, it will be used to characterize the method of merging the currently existing second specified shard according to the target merging method and the method of reducing the at least one virtual node corresponding to the target merged shard, as the merging method required for the target shard; Otherwise, the step of designating the merged fragment as the second specified fragment to be merged is triggered.

12. A data processing apparatus, characterized in that, include: The first determining module is used to determine each piece of target data from the data stored in the target storage system; wherein each piece of target data is data that meets predetermined data prediction conditions; the predetermined data prediction conditions include a first prediction condition for predicting hot data, or a second prediction condition for predicting cold data. The second determining module is used to determine the target fragment in the hash ring set by the target storage system based on each target data; wherein the target fragment is a fragment used to map at least one target data. An acquisition module is used to acquire an adjustment strategy for the target shard; wherein the adjustment strategy is used to indicate the adjustment method of the target shard and the target node corresponding to the shard obtained after adjustment; An execution module is used to execute the adjustment strategy on the target shard to adjust the target shard according to the adjustment method, and to map the at least one target data to its corresponding target node; wherein, the target node corresponding to each target data is the target node corresponding to the shard to which the target data needs to be mapped.

13. An electronic device, characterized in that, It includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, when executing a program stored in memory, implements the method described in any one of claims 1-11.

14. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method described in any one of claims 1-11.

Images