Method and apparatus for adjusting replication factor

Abstract

This application provides a method and apparatus for adjusting the replication factor. The method includes: acquiring multiple data segments stored in a distributed storage system; performing clustering processing on the multiple data segments to obtain clustering results; generating a first input feature vector for a target adjustment model corresponding to the distributed storage system based on data segments in multiple target groups and multiple index parameters of the distributed storage system; inputting the first input feature vector into the target adjustment model to analyze and adjust the initial replication factor of the target groups to obtain the adjustment result of the initial replication factor, wherein the target adjustment model is established using historical first input feature vectors. This application solves the technical problem of low reliability in distributed storage systems, thereby achieving the technical effect of improving the reliability of distributed storage systems.

Description

Technical Field

[0001] This application relates to the field of computers, and more specifically, to a method and apparatus for adjusting the replication factor. Background Technology

[0002] In distributed storage systems, data replication is a key technology for improving data availability and system performance, and it can be achieved by setting a replication factor. Currently, a static replication strategy is used, which sets the same fixed replication factor for all data to replicate data in the distributed storage system. However, this method relies on manually set rules and is difficult to adapt to dynamically changing distributed storage system loads, leading to technical problems such as low reliability of the distributed storage system. Summary of the Invention

[0003] This application provides a method and apparatus for adjusting the replication factor, so as to at least solve the technical problem of low reliability of distributed storage systems in related technologies.

[0004] According to one embodiment of this application, a method for adjusting the replication factor is provided, comprising: acquiring multiple data fragments stored in a distributed storage system; performing clustering processing on the multiple data fragments to obtain clustering results of the data fragments, wherein the clustering results are used to characterize the successful aggregation of multiple data fragments into multiple target groups; generating a first input feature vector of a target adjustment model corresponding to the distributed storage system based on the data fragments in the multiple target groups and multiple index parameters of the distributed storage system, wherein the index parameters are used to characterize the performance and / or operating status of the distributed storage system; inputting the first input feature vector into the target adjustment model to analyze and adjust the initial replication factor of the target groups to obtain the adjustment result of the initial replication factor, wherein the target adjustment model is established using historical first input feature vectors, and the initial replication factor is used to characterize the number of times the data fragments of the target groups are replicated in the distributed storage system.

[0005] In an exemplary embodiment, a first input feature vector for a target adjustment model corresponding to the distributed storage system is generated based on data segments in multiple target groups and multiple index parameters of the distributed storage system. This includes: extracting features from the data segments in the multiple target groups to obtain a first feature extraction vector for the multiple target groups; extracting features from the multiple index parameters to obtain a second feature extraction vector for the multiple index parameters; and generating a first input feature vector based on the first and second feature extraction vectors.

[0006] In an exemplary embodiment, inputting a first input feature vector into a target adjustment model to analyze and adjust the initial replication factor of the target group to obtain the adjustment result of the initial replication factor includes: inputting the first input feature vector into a target decision sub-model of the target adjustment model for analysis to obtain a first output feature vector of the target decision sub-model, wherein the first output feature vector includes the adjustment probability value of the initial replication factor and the initial output feature vector, and the target decision sub-model is used to establish the model through historical first input feature vectors; generating a decision action for the initial replication factor in response to the adjustment probability value being greater than the adjustment probability threshold, wherein the decision action is used to characterize the decision to adjust the initial replication factor; and determining the adjustment result based on the decision action, the first input feature vector, and the initial output feature vector.

[0007] In an exemplary embodiment, determining the adjustment result based on a decision action, a first input feature vector, and an initial output feature vector includes: determining the initial output feature vector as the second input feature vector of the target adjustment sub-model of the target adjustment model based on the decision action; concatenating the first and second input feature vectors to obtain the target input feature vector of the target adjustment sub-model; and inputting the target input feature vector into the target adjustment sub-model to adjust the initial replication factor to obtain the adjustment result. The target adjustment sub-model is established using historical target input feature vectors. The target decision sub-model can be called a decision model. The target adjustment sub-model can be called an adjustment model.

[0008] In an exemplary embodiment, inputting the target input feature vector into the target adjustment sub-model and adjusting the initial replication factor to obtain the adjustment result includes: inputting the target input feature vector into the target adjustment sub-model to obtain a second output feature vector, wherein the second output feature vector includes the adjusted value of the replication factor; and adjusting the initial replication factor based on the adjusted value to obtain the adjustment result.

[0009] In one exemplary embodiment, adjusting the initial replication factor based on the adjustment value to obtain an adjustment result includes: updating the initial replication factor based on the adjustment value to obtain a target replication factor corresponding to the initial replication factor; and determining the adjustment result based on the target replication factor.

[0010] In an exemplary embodiment, the method further includes: obtaining an initial adjustment model of the distributed storage system and a historical first input feature vector, wherein the initial adjustment model includes an initial decision sub-model and an initial adjustment sub-model, the initial decision sub-model being established using a long short-term memory network and a feedforward neural network, and the initial adjustment sub-model being established using an embedding network, a multi-head attention network, and a feedforward neural network, and the historical first input feature vector being determined using multiple historical index parameters and multiple historical data segments of the distributed storage system; training the initial decision sub-model using the historical first input feature vector to obtain a target decision sub-model, and inputting the historical first input feature vector into the target decision sub-model for analysis to obtain a historical first output feature vector of the target decision sub-model, wherein the historical first output feature vector includes a... In a distributed storage system, the historical adjustment probability values ​​of historical replication factors in the historical target group, and the historical initial output feature vectors are considered. In response to a historical adjustment probability value exceeding an adjustment probability threshold, historical decision actions of the historical replication factors are generated. Based on these historical decision actions, the historical initial output feature vector is determined as the historical second input feature vector of the initial adjustment sub-model. The historical first input feature vector and the historical second input feature vector are concatenated to obtain the historical target input feature vector of the initial adjustment sub-model. The initial adjustment sub-model is then trained using this historical target input feature vector to obtain the target adjustment sub-model. Based on the target constraint function of the initial adjustment model, the target decision sub-model, and the target adjustment sub-model, the target adjustment model is determined. The target constraint function is used to constrain at least one historical index parameter of the distributed storage system.

[0011] According to another embodiment of this application, a replication factor adjustment device is provided, comprising: a first acquisition module for acquiring multiple data fragments stored in a distributed storage system; a second acquisition module for performing clustering processing on the multiple data fragments to obtain clustering results of the data fragments, wherein the clustering results are used to characterize the successful aggregation of multiple data fragments into multiple target groups; a generation module for generating a first input feature vector of a target adjustment model corresponding to the distributed storage system based on the data fragments in the multiple target groups and multiple index parameters of the distributed storage system, wherein the index parameters are used to characterize the performance and / or operating status of the distributed storage system; and a third acquisition module for inputting the first input feature vector into the target adjustment model to analyze and adjust the initial replication factor of the target group to obtain the adjustment result of the initial replication factor, wherein the target adjustment model is established using historical first input feature vectors, and the initial replication factor is used to characterize the number of times the data fragments of the target group are replicated in the distributed storage system.

[0012] According to yet another embodiment of this application, a computer-readable storage medium is also provided, in which a computer program is stored, wherein the computer program is configured to perform the steps in any of the above method embodiments when it is run.

[0013] According to yet another embodiment of this application, an electronic device is also provided, including a memory and a processor, wherein a computer program is stored in the memory and the processor is configured to run the computer program to perform the steps in any of the above method embodiments.

[0014] According to yet another embodiment of this application, a computer program product is also provided, including a computer program that, when executed by a processor, implements the steps in any of the above method embodiments.

[0015] This application first obtains multiple data fragments stored in a distributed storage system, then performs clustering processing on these data fragments to obtain clustering results representing the successful aggregation of multiple data fragments into multiple target groups. Then, based on the data fragments in the multiple target groups and multiple index parameters of the distributed storage system, a first input feature vector corresponding to the distributed storage system is generated. Finally, the first input feature vector can be input into the target adjustment model to analyze and adjust the initial replication factor of the target groups, thereby achieving the purpose of obtaining the adjustment result of the initial replication factor. Considering that after clustering multiple data fragments to obtain multiple target groups, a first input feature vector is generated based on the data fragments in the multiple target groups and multiple index parameters of the distributed storage system, and this first input feature vector is input into the target adjustment model, the initial replication factor of the target groups is analyzed and adjusted to obtain a replication factor suitable for the target groups. Based on the obtained replication factor, data fragments in the target groups are replicated in the distributed storage system. This enables timely retrieval of data fragments from other nodes when a data fragment is detected to be lost in a certain node, thereby solving the technical problem of low reliability in distributed storage systems and achieving the technical effect of improving the reliability of distributed storage systems. Attached Figure Description

[0016] Figure 1 This is a hardware structure block diagram of a server device for a method of adjusting the replication factor according to an embodiment of this application.

[0017] Figure 2 This is a flowchart of a method for adjusting the replication factor according to an embodiment of this application;

[0018] Figure 3 This is a flowchart of a data replication factor adjustment method according to an embodiment of this application;

[0019] Figure 4 This is a schematic diagram of a dual-model structure according to an embodiment of this application;

[0020] Figure 5 This is a structural block diagram of a replication factor adjustment device according to an embodiment of this application. Detailed Implementation

[0021] The embodiments of this application will be described in detail below with reference to the accompanying drawings and examples.

[0022] It should be noted that the terms "first," "second," etc., in the specification, claims, and drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0023] The methods and embodiments provided in this application can be executed on a server device or a similar computing device. Taking running on a server device as an example, Figure 1 This is a hardware structure block diagram of a server device for a method of adjusting the replication factor according to an embodiment of this application. Figure 1 As shown, the server device may include one or more ( Figure 1 Only one is shown in the diagram. A processor 102 (which may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.) and a memory 104 for storing data are also shown. The server device may further include a transmission device 106 for communication functions and an input / output device 108. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the server equipment described above. For example, the server equipment may also include components that are more... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0024] The memory 104 can be used to store computer programs, such as application software programs and modules, like the computer program corresponding to the replication factor adjustment method in this embodiment. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, thus implementing the above-described method. The memory 104 may include high-speed random access memory and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to server devices via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0025] The transmission device 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by a communication provider for the server device. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 may be a Radio Frequency (RF) module used for wireless communication with the Internet.

[0026] This embodiment provides a method for adjusting the replication factor. Figure 2 This is a flowchart of a method for adjusting the replication factor according to an embodiment of this application, such as... Figure 2 As shown, the process includes the following steps:

[0027] Step S202: Obtain multiple data fragments stored in the distributed storage system.

[0028] In step S202 of this application embodiment, the access interface provided by the distributed storage system can be used to initiate a data read request to the distributed storage system in order to obtain multiple data fragments stored in the distributed storage system.

[0029] Alternatively, a distributed storage system is a computer system that stores data across multiple nodes to achieve efficient data management and access within a network environment; it can be simply referred to as a system. Data fragments may include attributes such as data size, data access frequency, and data importance.

[0030] It should be noted that this is only a preferred implementation method for obtaining multiple data fragments stored in a distributed storage system, and the process and method for obtaining multiple data fragments stored in a distributed storage system are not specifically limited.

[0031] Step S204: Perform clustering processing on multiple data segments to obtain the clustering results of the data segments.

[0032] In step S204 of this embodiment, a clustering algorithm (K-means algorithm) is used to cluster the obtained multiple data segments to obtain clustering results. These clustering results characterize the successful aggregation of multiple data segments into multiple target groups. The attribute information of the target groups includes: average size, average access frequency, average importance, and number of data segments. The clustering results can be referred to as data aggregation results.

[0033] Optionally, the K-means algorithm can be used to cluster the data segments in the system. Each data segment contains attributes such as data size, data access frequency, and data importance. Through clustering algorithms, a large number of independent data segments are grouped into a small number of data groups, significantly reducing the amount of data to be processed while preserving important statistical features. For example, 10,000 data segments can be aggregated into 10 groups, each group containing attributes such as average size, average access frequency, average importance, and the number of data segments. The features corresponding to these attributes are called group-level features.

[0034] It should be noted that this is only a preferred implementation method for obtaining the clustering results of data fragments. The process and method for obtaining the clustering results of data fragments are not specifically limited. As long as multiple data fragments are clustered, the process and method for obtaining the clustering results of the data fragments are within the protection scope of this application, and will not be listed here.

[0035] Step S206: Based on the data fragments in multiple target groups and multiple index parameters of the distributed storage system, generate the first input feature vector of the target adjustment model corresponding to the distributed storage system.

[0036] In step S206 of this application embodiment, after obtaining multiple target groups, a first input feature vector of the target adjustment model corresponding to the distributed storage system is generated based on the data fragments in the multiple target groups and multiple index parameters of the distributed storage system. The index parameters are used to characterize the performance and / or operating status of the distributed storage system and can be simply referred to as indexes.

[0037] Optionally, monitoring tools can be used to continuously monitor and collect various metrics of the distributed storage system, such as storage utilization, average replication factor, system load, request latency, system throughput, proportion of hot data, read-write ratio, node failure rate, data recovery time, cross-region traffic, storage space and bandwidth costs, etc. These metrics can comprehensively reflect the system's operating status and performance, and the characteristics corresponding to these metrics can be called system-level characteristics.

[0038] It is understood that this is only a preferred implementation of obtaining the first input feature vector of the target adjustment model, and the process and method of obtaining the first input feature vector of the target adjustment model are not specifically limited. As long as the process and method of generating the first input feature vector of the target adjustment model corresponding to the distributed storage system are based on data fragments in multiple target groups and multiple index parameters of the distributed storage system, they are all within the protection scope of this application and will not be listed here.

[0039] Step S208: Input the first input feature vector into the target adjustment model to analyze and adjust the initial replication factor of the target group, and obtain the adjustment result of the initial replication factor.

[0040] In step S208 of this application embodiment, the obtained first input feature vector is input into the target adjustment model to analyze and adjust the initial replication factor of the target group, so as to obtain the adjustment result of the initial replication factor.

[0041] Optionally, the target adjustment model is established using the historical first input feature vector. The initial replication factor is used to characterize the number of times the data fragments of the target group are replicated in the distributed storage system. It can be represented by a numerical value, such as 1, 3, etc. When the replication factor is 3, it is used to characterize that each piece of data has 3 copies in the system.

[0042] It should be noted that this is only a preferred embodiment for obtaining the adjustment result of the initial replication factor. The process and method for obtaining the adjustment result of the initial replication factor are not specifically limited. As long as the first input feature vector is input into the target adjustment model, the initial replication factor of the target group is analyzed and adjusted, and the process and method for obtaining the adjustment result of the initial replication factor are within the protection scope of this application, and will not be elaborated here.

[0043] Through the above steps, multiple data fragments stored in the distributed storage system are first obtained. These fragments are then clustered to obtain clustering results representing the successful aggregation of multiple data fragments into multiple target groups. Next, based on the data fragments in the target groups and multiple index parameters of the distributed storage system, a first input feature vector corresponding to the target adjustment model of the distributed storage system is generated. This first input feature vector is then input into the target adjustment model to analyze and adjust the initial replication factor of the target groups, thereby achieving the goal of obtaining the adjusted initial replication factor. Considering that after clustering multiple data fragments to obtain multiple target groups, a first input feature vector is generated based on the data fragments in the target groups and multiple index parameters of the distributed storage system, and this first input feature vector is input into the target adjustment model, the initial replication factor of the target groups is analyzed and adjusted to obtain a replication factor suitable for the target groups. Based on the obtained replication factor, data fragments in the target groups are replicated in the distributed storage system. This allows for timely retrieval of data fragments from other nodes when a data fragment is detected to be lost in a node, thus solving the technical problem of low reliability in distributed storage systems and achieving the technical effect of improving the reliability of distributed storage systems.

[0044] As an optional implementation, a first input feature vector for a target adjustment model corresponding to the distributed storage system is generated based on data segments in multiple target groups and multiple index parameters of the distributed storage system. This includes: extracting features from the data segments in multiple target groups to obtain a first feature extraction vector for the multiple target groups; extracting features from the multiple index parameters to obtain a second feature extraction vector for the multiple index parameters; and generating a first input feature vector based on the first and second feature extraction vectors.

[0045] In this embodiment, after obtaining multiple target groups and multiple indicator parameters, feature extraction can be performed on the data segments in the multiple target groups to obtain the first feature extraction vector of the multiple target groups. Then, feature extraction needs to be performed on the multiple indicator parameters to obtain the second feature extraction vector of the multiple indicator parameters. Based on the first feature extraction vector and the second feature extraction vector obtained above, the purpose of generating the first input feature vector can be achieved.

[0046] Optionally, the first feature extraction vector can be called a group-level feature vector. The second feature extraction vector can be called a system-level feature vector. The first input feature vector can be a feature vector containing 52 elements. It should be noted that this is only an example of the dimension of the first input feature vector, and the dimension of the first input feature vector is not specifically limited.

[0047] For example, a first input feature vector is constructed based on system-level and group-level feature vectors. This vector contains the collected system-level features and the collected group-level features for each data group (average size, access frequency, importance, and number of data fragments), so as to finally generate a feature vector containing 52 elements (12 system-level features + 10 group * 4 group-level features), improving the comprehensiveness and accuracy of the feature description of the performance of the distributed storage system, and achieving the effect of accurate analysis of the first input feature vector.

[0048] As an optional embodiment, the first input feature vector is input into the target adjustment model to analyze and adjust the initial replication factor of the target group, thereby obtaining the adjustment result of the initial replication factor. This includes: inputting the first input feature vector into the target decision sub-model of the target adjustment model for analysis, obtaining the first output feature vector of the target decision sub-model, wherein the first output feature vector includes the adjustment probability value of the initial replication factor and the initial output feature vector, and the target decision sub-model is established using historical first input feature vectors; in response to the adjustment probability value being greater than the adjustment probability threshold, generating a decision action for the initial replication factor, wherein the decision action characterizes the decision to adjust the initial replication factor; and determining the adjustment result based on the decision action, the first input feature vector, and the initial output feature vector.

[0049] In this embodiment, the obtained first input feature vector can be input into the target decision sub-model of the target adjustment model for analysis, so as to obtain the first output feature vector of the target decision sub-model. The first output feature vector may include the adjustment probability value of the initial replication factor and the initial output feature vector. The adjustment probability value is compared with the adjustment probability threshold. If the adjustment probability value is greater than the adjustment probability threshold, the decision action of the initial replication factor can be generated. Then, based on the decision action, the first input feature vector and the initial output feature vector, the adjustment result of the initial replication factor is determined.

[0050] Optionally, the adjustment probability threshold can be a preset threshold, such as 0.5. The target decision sub-model can be represented by a decision-model. The initial output feature vector can be represented by a feature vector v. The first output feature vector can be a data feature vector with dimensions (1, 41), where the output value of the first data is between 0 and 1, used to represent the probability of whether the replication factor needs to be adjusted. If the output value is greater than the preset threshold of 0.5, it is considered that adjustment is needed; if the output value is less than or equal to the preset threshold of 0.5, it is considered that the replication factor does not need to be adjusted. The values ​​of the remaining 40 dimensions are called feature vector v, which serves as part of the input of the target adjustment sub-model of the target adjustment model, thereby accurately determining whether the replication factor needs to be adjusted at this time, achieving the effect of accurately determining the adjustment result.

[0051] As an optional implementation, determining the adjustment result based on the decision action, the first input feature vector, and the initial output feature vector includes: determining the initial output feature vector as the second input feature vector of the target adjustment sub-model of the target adjustment model based on the decision action; concatenating the first input feature vector and the second input feature vector to obtain the target input feature vector of the target adjustment sub-model; inputting the target input feature vector into the target adjustment sub-model to adjust the initial replication factor to obtain the adjustment result, wherein the target adjustment sub-model is used to establish the model through historical target input feature vectors.

[0052] In this embodiment, based on the decision action used to characterize the adjustment of the initial replication factor, the initial output feature vector can be used to determine the second input feature vector of the target adjustment sub-model of the target adjustment model. Then, the first and second input feature vectors are concatenated to obtain the target input feature vector of the target adjustment sub-model. This obtained target input feature vector is then input into the target adjustment sub-model to adjust the initial replication factor and achieve the desired adjustment result. The target adjustment sub-model can be represented using an adjust-model.

[0053] For example, if the decision-model determines that the initial replication factor needs to be adjusted, the feature vector v and the output of the decision-model are concatenated and then input into the adjust-model. The adjust-model uses a neural network structure with a multi-head attention mechanism, which effectively extracts important features from the input features. This enriches the performance features of the distributed storage system while retaining important feature information, thus achieving the technical effect of effectively adjusting the initial replication factor.

[0054] As an optional implementation, the target input feature vector is input into the target adjustment sub-model to adjust the initial replication factor and obtain the adjustment result. This includes: inputting the target input feature vector into the target adjustment sub-model to obtain a second output feature vector, wherein the second output feature vector includes the adjustment value of the replication factor; and adjusting the initial replication factor based on the adjustment value to obtain the adjustment result.

[0055] In this embodiment, after obtaining the target input feature vector, the target input feature vector can be input into the target adjustment sub-model to obtain the adjustment value of the replication factor in the second output feature vector. Then, based on the adjustment value obtained above, the initial replication factor can be adjusted to obtain the adjustment result.

[0056] Optionally, the adjustment value is used to characterize the adjustment ratio of the replication factor for each data group, and ranges from -1 to 1, also representing the degree to which the replication factor decreases or increases. The second output feature vector has a dimension of [n, 10], where n represents the number of all nodes in the distributed storage system, and 10 represents the adjustment value of the replication factor for each data group in each node, such as 0.1, 0.5, etc.

[0057] As an optional implementation method, the initial replication factor is adjusted based on the adjustment value to obtain the adjustment result, including: updating the initial replication factor based on the adjustment value to obtain a target replication factor corresponding to the initial replication factor; and determining the adjustment result based on the target replication factor.

[0058] In this embodiment, after obtaining the adjustment value, the initial replication factor can be updated to obtain the target replication factor. Then, based on the obtained target replication factor, the adjustment result of the initial replication factor can be determined. The initial replication factor can be the current replication factor.

[0059] Optionally, the preset base value and the adjustment value are summed to obtain the target sum value, wherein the preset base value can be a preset value, such as 1; the target sum value is multiplied by the initial replication factor to obtain the target replication factor.

[0060] For example, if the current replication factor of a certain set of data is 2 and the adjustment value is 0.5, then the new replication factor will be 3, calculated as 2*(1+0.5)=3. The system will then perform actual data replication or deletion operations accordingly, thereby selecting appropriate replication factors for different data segments based on the current performance of the distributed storage system. This allows each data segment to adapt to the dynamically changing load of the distributed storage system, thereby improving the space utilization of the distributed storage system.

[0061] As an optional embodiment, the method further includes: obtaining an initial adjustment model of the distributed storage system and a historical first input feature vector, wherein the initial adjustment model includes an initial decision sub-model and an initial adjustment sub-model, the initial decision sub-model being established using a long short-term memory network and a feedforward neural network, and the initial adjustment sub-model being established using an embedding network, a multi-head attention network, and a feedforward neural network, and the historical first input feature vector being determined using multiple historical index parameters and multiple historical data segments of the distributed storage system; training the initial decision sub-model using the historical first input feature vector to obtain a target decision sub-model, and inputting the historical first input feature vector into the target decision sub-model for analysis to obtain the historical first output feature vector of the target decision sub-model, wherein the historical first output feature vector includes In a distributed storage system, the historical adjustment probability values ​​of historical replication factors in the historical target group, and the historical initial output feature vectors are considered. In response to a historical adjustment probability value exceeding an adjustment probability threshold, historical decision actions of the historical replication factors are generated. Based on these historical decision actions, the historical initial output feature vector is determined as the historical second input feature vector of the initial adjustment sub-model. The historical first input feature vector and the historical second input feature vector are concatenated to obtain the historical target input feature vector of the initial adjustment sub-model. The initial adjustment sub-model is then trained using this historical target input feature vector to obtain the target adjustment sub-model. Based on the target constraint function of the initial adjustment model, the target decision sub-model, and the target adjustment sub-model, the target adjustment model is determined. The target constraint function is used to constrain at least one historical index parameter of the distributed storage system.

[0062] In this embodiment, an initial adjustment model of the distributed storage system and a historical first input feature vector can be obtained first. The initial adjustment model includes an initial decision sub-model and an initial adjustment sub-model. The initial decision sub-model is established using a long short-term memory network and a feedforward neural network, while the initial adjustment sub-model is established using an embedding network, a multi-head attention network, and a feedforward neural network. Then, the initial decision sub-model can be trained using the historical first input feature vector to obtain a target decision sub-model. The historical first input feature vector is then input into the target decision sub-model for analysis to obtain the historical first output feature vector of the target decision sub-model. At this point, the historical first output feature vector includes the historical adjustment probability value of the historical replication factor in the historical target group and the historical initial output feature vector. The historical adjustment probability value is compared with the adjustment probability threshold. When the historical adjustment probability value is greater than the adjustment probability threshold, a historical decision action for the historical replication factor can be generated, indicating that the historical replication factor in the historical group needs to be adjusted. Further, based on the historical decision action obtained above, the historical initial output feature vector can be determined as the historical second input feature vector of the initial adjustment sub-model. Then, the obtained historical first input feature vector and historical second input feature vector are concatenated to obtain the historical target input feature vector of the initial adjustment sub-model. Using the historical target input feature vector, the initial adjustment sub-model is trained to obtain the target adjustment sub-model. Finally, based on the target constraint function of the initial adjustment model, the target decision sub-model, and the target adjustment sub-model, the target adjustment model can be determined. The target adjustment model can be called a dual model.

[0063] Optionally, after obtaining the target decision sub-model and the target adjustment sub-model, the target decision sub-model and the target adjustment sub-model are iteratively trained using the target constraint function to obtain a trained model, which is the target adjustment model. The target constraint function can be a reward function, and can be composed of target throughput, target storage utilization, and target cost rate, and can be represented using R.

[0064] Optionally, the historical throughput, current throughput, historical storage utilization, current storage utilization, historical total cost, and current total cost of the distributed storage system are obtained; a target throughput rate is determined based on the historical throughput and current throughput; a target storage utilization rate is determined based on the historical storage utilization and current storage utilization; a target cost rate is determined based on the historical total cost and current total cost; and a target constraint function is determined based on the target throughput rate, target storage utilization rate, and target cost rate.

[0065] Furthermore, subtracting historical throughput from current throughput yields the first difference. The ratio of this first difference to historical throughput is determined as the target throughput. Here, current throughput can be called the new throughput, and historical throughput can be called the old throughput. The target throughput can be represented by ΔPerformance. Subtracting historical storage utilization from current storage utilization yields the second difference. The ratio of this second difference to historical storage utilization is determined as the target storage utilization rate. Here, current storage utilization can be called the new storage utilization value, and historical storage utilization can be called the old storage utilization value. The target storage utilization rate can be represented by ΔResourceUtilization. Subtracting historical total cost from current total cost yields the third difference. The ratio of this third difference to historical total cost is determined as the target cost rate. Here, current total cost can be called the new total cost, and historical total cost can be called the old total cost. The target cost rate can be represented by ΔCost.

[0066] For example, reinforcement learning (RL) is used to continuously optimize the decision-model and adjust-model. The purpose of using this model is to optimize the performance of the distributed storage system in order to improve its availability. Therefore, when defining the reward function, factors such as performance improvement, resource utilization optimization, and cost reduction need to be considered. The reward function can then be determined using the following formula:

[0067] R=w1ΔPerformance+w2ΔResourceUtilizattion-w3ΔCost

[0068] From the above formulas, we know that ΔPerformance = (New Throughput - Old Throughput) / Old Throughput, ΔResourceUtilization = (New Storage Utilization - Old Storage Utilization) / Old Storage Utilization, ΔCost = (New Total Cost - Old Total Cost) / Old Total Cost, where w1, w2, and w3 represent weight coefficients, and their sum is 1. It should be noted that a pre-defined framework (actor-critic framework) is used during training. In this framework, the actor represents the combination of the decision-model and the adjust-model, and the critic is a network specifically used to train the actor. The critic can be used to evaluate the impact of the two models on the output actions after execution.

[0069] Furthermore, the training involves two models: the decision-model and the adjust-model, which need to be trained separately. Both use the temporal difference error (TDerror) as a guide to update the model gradients. The difference lies in that some outputs of the decision-model are applied to the adjust-model. Therefore, the adjust-model updates once per update, while the decision-model updates twice, and itself updates once, meaning it updates along with the adjust-model. During training, experience replay is used in a buffer to store historical decisions and their results. These experiences are periodically used to update the model parameters, and a noise mechanism is used to improve training effectiveness and increase model robustness.

[0070] Understandably, iterative model training is necessary. After each round of adjustments, the system continues to collect new state information to prepare for the next round of decision-making. This process is ongoing to ensure the system can continuously adapt to environmental changes. After continuous training, the model converges and can be formally deployed in a distributed storage system. During use, at regular intervals, the system-level and group-level features of each node in the entire system are input into the decision-model. Then, it is determined whether to modify the replication factor. If necessary, the adjust-model is used to determine the size of the modification factor and the nodes to be modified, so that the replication factor modification operation can be executed.

[0071] In this embodiment, multiple data fragments stored in the distributed storage system are first acquired. These fragments are then clustered to obtain clustering results representing the successful aggregation of multiple data fragments into multiple target groups. Based on the data fragments in the target groups and multiple index parameters of the distributed storage system, a first input feature vector corresponding to the distributed storage system is generated. This first input feature vector is then input into the target adjustment model to analyze and adjust the initial replication factor of the target groups, thereby achieving the goal of obtaining the adjustment result of the initial replication factor. Considering that after clustering multiple data fragments to obtain multiple target groups, a first input feature vector is generated based on the data fragments in the target groups and multiple index parameters of the distributed storage system, and this first input feature vector is input into the target adjustment model, the initial replication factor of the target groups is analyzed and adjusted to obtain a replication factor suitable for the target groups. Based on the obtained replication factor, data fragments in the target groups are replicated in the distributed storage system. This allows for timely retrieval of data fragments from other nodes when a data fragment is detected to be lost in a node, thus solving the technical problem of low reliability in distributed storage systems and achieving the technical effect of improving the reliability of distributed storage systems.

[0072] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods of the various embodiments of this application.

[0073] The technical solutions of the embodiments of the present invention will be illustrated below with reference to preferred embodiments.

[0074] In distributed storage systems, data replication is a key technology for improving data availability and system performance, and it can be achieved by setting a replication factor. Data replication refers to storing copies of the same data on multiple storage nodes, while the replication factor indicates the number of times each piece of data is copied. For example, a replication factor of 3 means that each piece of data has 3 copies in the system.

[0075] Currently, data replication in distributed storage systems is achieved by using a static replication strategy, which sets the same fixed replication factor for all data. However, this method relies on manually set rules, lacks flexibility and adaptability, and is difficult to adapt to dynamically changing distributed storage system loads, resulting in technical problems such as low reliability of distributed storage systems.

[0076] To address the aforementioned issues, a reinforcement learning-based dynamic replication factor adjustment method is proposed. This method employs a dual-model structure to achieve intelligent and automated adjustment of the replication factor in distributed storage systems. The method includes a decision model for modifying the replication factor and a dynamic adjustment model. Based on these two models, an adaptive reinforcement learning framework is constructed. This framework dynamically determines the replication factor in the distributed storage system by analyzing system information such as storage utilization and storage node load. Furthermore, the model can balance data availability, system performance, and storage overhead, outputting the optimal replication factor configuration under different system loads and data characteristics. This solves the technical problem of low reliability in distributed storage systems and achieves the technical effect of improving the reliability of distributed storage systems.

[0077] Figure 3 This is a flowchart of a data replication factor adjustment method according to an embodiment of this application, such as... Figure 3 As shown, the method mainly includes the following steps:

[0078] Step S301: Obtain system information of the distributed storage system.

[0079] In this embodiment, system information of the distributed storage system can be obtained. Monitoring tools are used to continuously monitor and collect various metrics of the distributed storage system, such as storage utilization, average replication factor, system load, request latency, system throughput, proportion of hot data, read-write ratio, node failure rate, data recovery time, cross-region traffic, storage space and bandwidth costs, etc. These metrics can comprehensively reflect the system's operating status and performance, and the characteristics corresponding to these metrics can be called system-level characteristics.

[0080] Step S302: Input system information into the decision model.

[0081] In this embodiment, multiple data fragments stored in the distributed storage system are obtained, and system information and multiple data fragments can be input into the decision model, which can be represented by a decision-model.

[0082] Optionally, the K-means algorithm can be used to cluster the data segments in the system. Each data segment contains attributes such as data size, data access frequency, and data importance. Through clustering algorithms, a large number of independent data segments are grouped into a small number of data groups, significantly reducing the amount of data to be processed while preserving important statistical features. For example, 10,000 data segments can be aggregated into 10 groups, each group containing attributes such as average size, average access frequency, average importance, and the number of data segments. The features corresponding to these attributes are called group-level features.

[0083] Furthermore, based on the system-level feature vector and the group-level feature vector, a first input feature vector is constructed. This vector contains the collected system-level features and the collected group-level features for each data group (average size, access frequency, importance, and number of data segments), so as to finally generate a feature vector containing 52 elements (12 system-level features + 10 group * 4 group-level features).

[0084] Optionally, the implementation process of the decision-model is as follows: the feature vector constructed by each node is input into the decision-model. Considering that the number of nodes in each distributed storage system is different, the decision-model adopts an LSTM and multilayer perceptron structure. In the initial stage of the model, an LSTM network is used to process each node to increase the availability of the model. The feature vector after feature extraction is input into the multilayer perceptron for processing, and finally outputs a data with dimensions (1, 41).

[0085] Step S303: Should the replication factor be adjusted?

[0086] In this embodiment, it is necessary to determine whether to adjust the replication factor. If yes, then step S304 is executed; otherwise, the process ends.

[0087] Optionally, the above steps can output a data set with dimensions (1, 41), where the first data point has a value between 0 and 1, representing the probability of needing to adjust the replication factor. If the output value is greater than a preset threshold (e.g., 0.5), adjustment is considered necessary. The values ​​of the remaining 40 dimensions are called the feature vector v, which serves as part of the input to the adjust-model.

[0088] Step S304: Generate decision action.

[0089] In this embodiment, if it is necessary to adjust the replication factor, a decision action is generated.

[0090] Step S305: Input the decision action and system information into the adjustment model.

[0091] In this embodiment, the decision actions and system information obtained from the above steps can be input into the adjustment model, and the adjustment model can be represented by the adjust-model.

[0092] Step S306: Determine the adjustment ratio of the replication factor.

[0093] In this embodiment, the adjustment ratio of the replication factor can be determined by adjusting the model. The implementation process of the adjust-model is as follows: if the decision-model determines that adjustment is needed, the feature vector v and the input vector of the decision-model are concatenated and then input into the adjust-model. The adjust-model adopts a neural network structure with a multi-head attention mechanism, so the dimension of the output data is [n, 10], where n represents the number of all nodes in the distributed storage system, and 10 represents the adjustment value of the replication factor for each data group in each node. The adjustment value is in the range of -1 to 1, such as 0.1, 0.5, etc.

[0094] Optionally, the replication factor of data segments in each data group can be updated based on the output of the adjust-model. For example, if the current replication factor of a data group is 2 and the adjustment value is 0.5, the new replication factor will be 3, calculated as 2*(1+0.5)=3. The system will then perform the actual data copying or deletion operation accordingly.

[0095] In this embodiment, the model needs to be trained. The training process can be as follows: reinforcement learning (RL) is used to continuously optimize the decision-model and adjust-model. The purpose of using this model is to optimize the performance of the distributed storage system in order to improve its availability. Therefore, when defining the reward function, factors such as system performance improvement, resource utilization optimization, and cost reduction need to be considered. The reward function can be determined by the following formula:

[0096] R=w1ΔPerformance+w2ΔResourceUtilization-w3ΔCost

[0097] From the above formulas, we know that ΔPerformance = (New Throughput - Old Throughput) / Old Throughput, ΔResourceUtilization = (New Storage Utilization - Old Storage Utilization) / Old Storage Utilization, ΔCost = (New Total Cost - Old Total Cost) / Old Total Cost, where w1, w2, and w3 represent weight coefficients, and their sum is 1. It should be noted that a pre-defined framework (actor-critic framework) is used during training. In this framework, the actor represents the combination of the decision-model and the adjust-model, and the critic is a network specifically used to train the actor. The critic can be used to evaluate the impact of the two models on the output actions after execution.

[0098] Furthermore, the training involves two models: the decision-model and the adjust-model, which need to be trained separately. Both use the temporal difference error (TDerror) as a guide to update the model gradients. The difference lies in that some outputs of the decision-model are applied to the adjust-model. Therefore, the adjust-model updates once per update, while the decision-model updates twice, and itself updates once, meaning it updates along with the adjust-model. During training, experience replay is used in a buffer to store historical decisions and their results. These experiences are periodically used to update the model parameters, and a noise mechanism is used to improve training effectiveness and increase model robustness.

[0099] Understandably, iterative model training is necessary. After each round of adjustments, the system continues to collect new state information to prepare for the next round of decision-making. This process is ongoing to ensure the system can continuously adapt to environmental changes. After continuous training, the model converges and can be formally deployed in a distributed storage system. During use, at regular intervals, the system-level and group-level features of each node in the entire system are input into the decision-model. Then, it is determined whether to modify the replication factor. If necessary, the adjust-model is used to determine the size of the modification factor and the nodes to be modified, so that the replication factor modification operation can be executed.

[0100] In this embodiment, after clustering multiple data segments to obtain multiple target groups, a first input feature vector is generated based on the data segments in the multiple target groups and multiple index parameters of the distributed storage system. This first input feature vector is then input into the target adjustment model to analyze and adjust the initial replication factor of the target groups, obtaining a replication factor suitable for the target groups. Based on the obtained replication factor, data segments in the target groups are replicated in the distributed storage system. This enables timely retrieval of data segments from other nodes when a data segment is detected to be lost in a certain node, thereby solving the technical problem of low reliability in the distributed storage system and achieving the technical effect of improving the reliability of the distributed storage system.

[0101] This application proposes a reinforcement learning-based method for dynamic adjustment of the replication factor. The method employs a dual-model structure, including a decision model and an adjustment model. The specific scheme is as follows: First, the system collects comprehensive state information, including indicators across multiple dimensions such as storage utilization, system load, and request latency. Then, the K-means algorithm is used to aggregate data fragments, reducing the decision space. Next, feature engineering is used to construct an input vector containing system-level features and data group features. The decision model determines whether the replication factor needs adjustment based on these features; if so, the adjustment model determines the specific adjustment scheme for each data group. Finally, the system updates the replication factor of the data fragments based on the model output, thereby solving the technical problem of low reliability in distributed storage systems and achieving the technical effect of improving the reliability of distributed storage systems.

[0102] Figure 4 This is a schematic diagram of a dual-model structure according to an embodiment of this application, such as... Figure 4As shown, the dual-model structure includes a decision model and an adjustment model. The decision model is constructed using a Long Short-Term Memory Network (LSTM) and a feedforward neural network (MLP), while the adjustment model can be constructed using an embedding network, a multi-head attention network, and a feedforward neural network.

[0103] Optionally, in the decision model, system information can be acquired first, and then the system information can be input into an LSTM network to obtain the first output feature. The first output feature can be input into a feedforward neural network to generate a decision action and a feature vector v. In the adjustment model, the output data in the decision model, i.e., the feature vector v and the acquired system information, can be input into an embedding network to obtain the second output feature. The second output feature can be input into a multi-head attention network to obtain the third output feature. The third output feature can be input into a feedforward neural network to obtain the replication factor.

[0104] This application proposes a reinforcement learning-based method for dynamic adjustment of the replication factor. The method employs a dual-model structure, including a decision model and an adjustment model. The specific scheme is as follows: First, the system collects comprehensive state information, including indicators across multiple dimensions such as storage utilization, system load, and request latency. Then, the K-means algorithm is used to aggregate data fragments, reducing the decision space. Next, feature engineering is used to construct an input vector containing system-level features and data group features. The decision model determines whether the replication factor needs adjustment based on these features; if so, the adjustment model determines the specific adjustment scheme for each data group. Finally, the system updates the replication factor of the data fragments based on the model output, thereby solving the technical problem of low reliability in distributed storage systems and achieving the technical effect of improving the reliability of distributed storage systems.

[0105] This embodiment also provides a replication factor adjustment device for implementing the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can refer to a combination of software and / or hardware that performs a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0106] Figure 5 This is a structural block diagram of a replication factor adjustment device according to an embodiment of this application, such as... Figure 5 As shown, the device includes: a first acquisition module 501, a second acquisition module 502, a generation module 503, and a third acquisition module 504.

[0107] The first acquisition module 501 is used to acquire multiple data fragments stored in the distributed storage system.

[0108] The second acquisition module 502 is used to perform clustering processing on multiple data segments to obtain clustering results of the data segments, wherein the clustering results are used to characterize the successful aggregation of multiple data segments into multiple target groups.

[0109] The generation module 503 is used to generate a first input feature vector of the target adjustment model corresponding to the distributed storage system based on data fragments in multiple target groups and multiple index parameters of the distributed storage system. The index parameters are used to characterize the performance and / or operating status of the distributed storage system.

[0110] The third acquisition module 504 is used to input the first input feature vector into the target adjustment model, analyze and adjust the initial replication factor of the target group, and obtain the adjustment result of the initial replication factor. The target adjustment model is established by using the historical first input feature vector, and the initial replication factor is used to characterize the number of times the data fragments of the target group are replicated in the distributed storage system.

[0111] Optionally, the generation module 503 is further configured to perform feature extraction on the data segments in the multiple target groups respectively to obtain a first feature extraction vector of the multiple target groups; perform feature extraction on the multiple indicator parameters respectively to obtain a second feature extraction vector of the multiple indicator parameters; and generate a first input feature vector based on the first feature extraction vector and the second feature extraction vector.

[0112] Optionally, the third acquisition module 504 is further configured to input the first input feature vector into the target decision sub-model of the target adjustment model for analysis, and obtain the first output feature vector of the target decision sub-model, wherein the first output feature vector includes the adjustment probability value of the initial replication factor and the initial output feature vector, and the target decision sub-model is established by using the historical first input feature vector; in response to the adjustment probability value being greater than the adjustment probability threshold, a decision action for the initial replication factor is generated, wherein the decision action is used to characterize the decision to adjust the initial replication factor; and the adjustment result is determined based on the decision action, the first input feature vector and the initial output feature vector.

[0113] Optionally, the third acquisition module 504 is further configured to, based on the decision action, determine the initial output feature vector as the second input feature vector of the target adjustment sub-model of the target adjustment model; concatenate the first input feature vector and the second input feature vector to obtain the target input feature vector of the target adjustment sub-model; input the target input feature vector into the target adjustment sub-model, adjust the initial replication factor, and obtain the adjustment result, wherein the target adjustment sub-model is established by using historical target input feature vectors.

[0114] Optionally, the third acquisition module 504 is further configured to input the target input feature vector into the target adjustment sub-model to obtain a second output feature vector, wherein the second output feature vector includes the adjustment value of the replication factor; and adjust the initial replication factor based on the adjustment value to obtain the adjustment result.

[0115] Optionally, the third acquisition module 504 is further configured to update the initial replication factor based on the adjustment value to obtain the target replication factor corresponding to the initial replication factor; and determine the adjustment result based on the target replication factor.

[0116] Optionally, the device further includes: a fourth acquisition module, used to acquire the initial adjustment model of the distributed storage system and the historical first input feature vector, wherein the initial adjustment model includes an initial decision sub-model and an initial adjustment sub-model, the initial decision sub-model is established using a long short-term memory network and a feedforward neural network, the initial adjustment sub-model is established using an embedding network, a multi-head attention network, and a feedforward neural network, and the historical first input feature vector is determined using multiple historical index parameters and multiple historical data segments of the distributed storage system; a fifth acquisition module, used to train the initial decision sub-model using the historical first input feature vector to obtain a target decision sub-model, and input the historical first input feature vector into the target decision sub-model for analysis to obtain the historical first output feature vector of the target decision sub-model, wherein the historical first output feature vector includes data from the distributed storage system. The system comprises: a historical adjustment probability value of the historical replication factor in the historical target group, and a historical initial output feature vector; a first determining module, used to generate historical decision actions of the historical replication factor in response to the historical adjustment probability value being greater than the adjustment probability threshold, and based on the historical decision actions, determine the historical initial output feature vector as the historical second input feature vector of the initial adjustment sub-model; a sixth obtaining module, used to concatenate the historical first input feature vector and the historical second input feature vector to obtain the historical target input feature vector of the initial adjustment sub-model, and use the historical target input feature vector to train the initial adjustment sub-model to obtain the target adjustment sub-model; and a second determining module, used to determine the target adjustment model based on the target constraint function of the initial adjustment model, the target decision sub-model, and the target adjustment sub-model, wherein the target constraint function is used to constrain at least one historical index parameter of the distributed storage system.

[0117] In this embodiment, a first acquisition module acquires multiple data fragments stored in the distributed storage system; a second acquisition module performs clustering processing on the multiple data fragments to obtain clustering results, wherein the clustering results are used to characterize the successful aggregation of multiple data fragments into multiple target groups; a generation module generates a first input feature vector for a target adjustment model corresponding to the distributed storage system based on the data fragments in the multiple target groups and multiple indicator parameters of the distributed storage system, wherein the indicator parameters are used to characterize the performance and / or operating status of the distributed storage system; a third acquisition module inputs the first input feature vector into the target adjustment model to analyze and adjust the initial replication factor of the target group to obtain the adjustment result of the initial replication factor, wherein the target adjustment model is established using historical first input feature vectors, and the initial replication factor is used to characterize the number of times the data fragments of the target group are replicated in the distributed storage system, thereby solving the technical problem of low reliability of the distributed storage system and achieving the technical effect of improving the reliability of the distributed storage system.

[0118] It should be noted that the above modules can be implemented by software or hardware. For the latter, they can be implemented in the following ways, but are not limited to: all the above modules are located in the same processor; or, the above modules are located in different processors in any combination.

[0119] Embodiments of this application also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above method embodiments when run.

[0120] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0121] Embodiments of this application also provide an electronic device, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the steps in any of the above method embodiments.

[0122] In one exemplary embodiment, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor and the input / output device is connected to the processor.

[0123] Embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above method embodiments.

[0124] Embodiments of this application also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps in any of the above method embodiments.

[0125] The embodiments described herein also provide a computer program that includes computer instructions stored in a computer-readable storage medium; a processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the steps in any of the above method embodiments.

[0126] Specific examples in this embodiment can be found in the examples described in the above embodiments and exemplary implementations, and will not be repeated here.

[0127] Obviously, those skilled in the art should understand that the modules or steps of this application described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. They can be implemented using computer-executable program code, and thus can be stored in a storage device for execution by a computing device. In some cases, the steps shown or described can be performed in a different order than those presented here, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, this application is not limited to any particular combination of hardware and software.

[0128] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the principles of this application should be included within the protection scope of this application.

Claims

1. A method for adjusting the replication factor, characterized in that, include: Retrieve multiple data fragments stored in a distributed storage system; Clustering is performed on multiple data segments to obtain clustering results for the data segments, wherein the clustering results are used to characterize the successful aggregation of multiple data segments into multiple target groups; Based on the data fragments in the multiple target groups and multiple index parameters of the distributed storage system, a first input feature vector of the target adjustment model corresponding to the distributed storage system is generated, wherein the index parameters are used to characterize the performance and / or operating status of the distributed storage system; The first input feature vector is input into the target adjustment model to analyze and adjust the initial replication factor of the target group, thereby obtaining the adjustment result of the initial replication factor. The target adjustment model is established using the historical first input feature vector, and the initial replication factor is used to characterize the number of times the data segment of the target group is replicated in the distributed storage system. The step of inputting the first input feature vector into the target adjustment model to analyze and adjust the initial replication factor of the target group to obtain the adjustment result of the initial replication factor includes: inputting the first input feature vector into the target decision sub-model of the target adjustment model for analysis to obtain the first output feature vector of the target decision sub-model, wherein the first output feature vector includes the adjustment probability value of the initial replication factor and the initial output feature vector, and the target decision sub-model is used to establish the model through the historical first input feature vector; in response to the adjustment probability value being greater than the adjustment probability threshold, generating a decision action for the initial replication factor, wherein the decision action is used to characterize the decision to adjust the initial replication factor; and determining the adjustment result based on the decision action, the first input feature vector, and the initial output feature vector. Determining the adjustment result based on the decision action, the first input feature vector, and the initial output feature vector includes: determining the initial output feature vector as the second input feature vector of the target adjustment sub-model of the target adjustment model based on the decision action; concatenating the first input feature vector and the second input feature vector to obtain the target input feature vector of the target adjustment sub-model; inputting the target input feature vector into the target adjustment sub-model to adjust the initial replication factor to obtain the adjustment result, wherein the target adjustment sub-model is established using historical target input feature vectors.

2. The method according to claim 1, characterized in that, The first input feature vector of the target adjustment model corresponding to the distributed storage system is generated based on the data fragments in the multiple target groups and multiple index parameters of the distributed storage system, including: Feature extraction is performed on the data segments in the multiple target groups respectively to obtain the first feature extraction vector of the multiple target groups; The feature extraction is performed on the multiple indicator parameters respectively to obtain the second feature extraction vector of the multiple indicator parameters; The first input feature vector is generated based on the first feature extraction vector and the second feature extraction vector.

3. The method according to claim 1, characterized in that, The step of inputting the target input feature vector into the target adjustment sub-model to adjust the initial replication factor and obtain the adjustment result includes: The target input feature vector is input into the target adjustment sub-model to obtain a second output feature vector, wherein the second output feature vector includes the adjustment value of the replication factor; Based on the adjustment value, the initial replication factor is adjusted to obtain the adjustment result.

4. The method according to claim 3, characterized in that, The adjustment of the initial replication factor based on the adjustment value to obtain the adjustment result includes: Based on the adjustment value, the initial replication factor is updated to obtain the target replication factor corresponding to the initial replication factor; The adjustment result is determined based on the target replication factor.

5. The method according to any one of claims 1 to 4, characterized in that, The method further includes: The initial adjustment model of the distributed storage system and the historical first input feature vector are obtained. The initial adjustment model includes an initial decision sub-model and an initial adjustment sub-model. The initial decision sub-model is established by using a long short-term memory network and a feedforward neural network. The initial adjustment sub-model is established by using an embedding network, a multi-head attention network and the feedforward neural network. The historical first input feature vector is determined by using multiple historical index parameters and multiple historical data fragments of the distributed storage system. The initial decision sub-model is trained using the historical first input feature vector to obtain the target decision sub-model. The historical first input feature vector is then input into the target decision sub-model for analysis to obtain the historical first output feature vector of the target decision sub-model. The historical first output feature vector includes the historical adjustment probability value of the historical replication factor in the historical target group in the distributed storage system, as well as the historical initial output feature vector. In response to the historical adjustment probability value being greater than the adjustment probability threshold, a historical decision action of the historical replication factor is generated, and based on the historical decision action, the historical initial output feature vector is determined as the historical second input feature vector of the initial adjustment sub-model; The historical first input feature vector and the historical second input feature vector are concatenated to obtain the historical target input feature vector of the initial adjustment sub-model. The initial adjustment sub-model is then trained using the historical target input feature vector to obtain the target adjustment sub-model. Based on the objective constraint function of the initial adjustment model, the objective decision sub-model, and the objective adjustment sub-model, the objective adjustment model is determined, wherein the objective constraint function is used to constrain at least one of the historical index parameters of the distributed storage system.

6. A device for adjusting the replication factor, characterized in that, include: The first acquisition module is used to acquire multiple data fragments stored in the distributed storage system; The second acquisition module is used to perform clustering processing on multiple data segments to obtain clustering results of the data segments, wherein the clustering results are used to characterize the successful aggregation of multiple data segments into multiple target groups; The generation module is used to generate a first input feature vector of the target adjustment model corresponding to the distributed storage system based on the data fragments in the multiple target groups and multiple indicator parameters of the distributed storage system, wherein the indicator parameters are used to characterize the performance and / or operating status of the distributed storage system; The third acquisition module is used to input the first input feature vector into the target adjustment model, analyze and adjust the initial replication factor of the target group, and obtain the adjustment result of the initial replication factor. The target adjustment model is used to establish the model through the historical first input feature vector, and the initial replication factor is used to characterize the number of times the data segment of the target group is replicated in the distributed storage system. The second acquisition module implements the following method: inputting the first input feature vector into the target adjustment model to analyze and adjust the initial replication factor of the target group, and obtaining the adjustment result of the initial replication factor: inputting the first input feature vector into the target decision sub-model of the target adjustment model for analysis, and obtaining the first output feature vector of the target decision sub-model, wherein the first output feature vector includes the adjustment probability value of the initial replication factor and the initial output feature vector, and the target decision sub-model is used to establish the model through the historical first input feature vector; in response to the adjustment probability value being greater than the adjustment probability threshold, generating a decision action for the initial replication factor, wherein the decision action is used to characterize the decision to adjust the initial replication factor; and determining the adjustment result based on the decision action, the first input feature vector, and the initial output feature vector. The second acquisition module determines the adjustment result based on the decision action, the first input feature vector, and the initial output feature vector in the following manner: based on the decision action, the initial output feature vector is determined as the second input feature vector of the target adjustment sub-model of the target adjustment model; the first input feature vector and the second input feature vector are concatenated to obtain the target input feature vector of the target adjustment sub-model; the target input feature vector is input into the target adjustment sub-model to adjust the initial replication factor and obtain the adjustment result, wherein the target adjustment sub-model is established using historical target input feature vectors.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, wherein the computer program, when executed by a processor, implements the steps of the method described in any one of claims 1 to 5.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method described in any one of claims 1 to 5.

Images