Replication factor adjusting method and device

Abstract

The embodiment of the invention provides a method and device for adjusting a replication factor, and the method comprises the steps: obtaining a plurality of data fragments stored in a distributed storage system; performing clustering processing on the plurality of data fragments to obtain a clustering result of the data fragments; generating a first input feature vector of a target adjustment model corresponding to the distributed storage system based on the data fragments in the plurality of target groups and a plurality of index parameters of the distributed storage system; the first input feature vector is input into a target adjustment model, the initial replication factor of the target group is analyzed and adjusted, an adjustment result of the initial replication factor is obtained, and the target adjustment model is established through historical first input feature vectors. Through the method and the device, the technical problem of low reliability of the distributed storage system is solved, and the technical effect of improving the reliability of the distributed storage system is further achieved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of computers, and more particularly, to a method and device for adjusting replication factors. Background Art

[0002] In a distributed storage system, data replication is a key technology for improving data availability and system performance, and can be achieved by setting replication factors. Currently, by adopting a static replication policy, that is, setting the same fixed replication factor for all data, data replication in the distributed storage system is realized. However, the above method depends on artificially set rules and is difficult to adapt to the dynamically changing load of the distributed storage system, resulting in the technical problem of low reliability of the distributed storage system. Summary of the Invention

[0003] The embodiments of the present application provide a method and device for adjusting replication factors to at least solve the technical problem of low reliability of the distributed storage system in the related art.

[0004] According to an embodiment of the present application, a method for adjusting a replication factor is provided, including: obtaining a plurality of data segments stored in a distributed storage system; performing clustering processing on the plurality of data segments to obtain a clustering result of the data segments, where the clustering result is used to represent successfully aggregating the plurality of data segments into a plurality of target groups; generating a first input feature vector of a target adjustment model corresponding to the distributed storage system based on the data segments in the plurality of target groups and a plurality of metric parameters of the distributed storage system, where the metric parameters are used to represent the performance and / or operating state of the distributed storage system; inputting the first input feature vector into the target adjustment model to analyze and adjust the initial replication factor of the target group to obtain an adjustment result of the initial replication factor, where the target adjustment model is established by using historical first input feature vectors, and the initial replication factor is used to represent the replication times of the data segments of the target group in the distributed storage system.

[0005] In an exemplary embodiment, generating a first input feature vector of a target adjustment model corresponding to the distributed storage system based on the data segments in the plurality of target groups and a plurality of metric parameters of the distributed storage system includes: respectively performing feature extraction on the data segments in the plurality of target groups to obtain first feature extraction vectors of the plurality of target groups; respectively performing feature extraction on the plurality of metric parameters to obtain second feature extraction vectors of the plurality of metric parameters; generating a first input feature vector based on the first feature extraction vector and the second feature extraction vector.

[0006] In an exemplary embodiment, a first input feature vector is input into a target adjustment model to analyze and adjust an initial replication factor of a target group, and an adjustment result of the initial replication factor is obtained, including: inputting the first input feature vector into a target decision sub-model of the target adjustment model for analysis to obtain a first output feature vector of the target decision sub-model, where the first output feature vector includes an adjustment probability value of the initial replication factor and an initial output feature vector, and the target decision sub-model is used to be established by historical first input feature vectors; in response to the adjustment probability value being greater than an adjustment probability threshold, generating a decision action for the initial replication factor, where the decision action is used to represent a decision to adjust the initial replication factor; and determining the adjustment result based on the decision action, the first input feature vector, and the initial output feature vector.

[0007] In an exemplary embodiment, determining the adjustment result based on the decision action, the first input feature vector, and the initial output feature vector includes: based on the decision action, determining the initial output feature vector as a second input feature vector of a target adjustment sub-model of the target adjustment model; performing a splicing process on the first input feature vector and the second input feature vector to obtain a target input feature vector of the target adjustment sub-model; inputting the target input feature vector into the target adjustment sub-model to adjust the initial replication factor to obtain an adjustment result, where the target adjustment sub-model is used to be established by historical target input feature vectors. The target decision sub-model can be referred to as a decision model. The target adjustment sub-model can be referred to as an adjustment model.

[0008] In an exemplary embodiment, inputting the target input feature vector into the target adjustment sub-model to adjust the initial replication factor to obtain an adjustment result includes: inputting the target input feature vector into the target adjustment sub-model to obtain a second output feature vector, where the second output feature vector includes an adjustment value of the replication factor; and adjusting the initial replication factor based on the adjustment value to obtain an adjustment result.

[0009] In an exemplary embodiment, adjusting the initial replication factor based on the adjustment value to obtain an adjustment result includes: updating the initial replication factor based on the adjustment value to obtain a target replication factor corresponding to the initial replication factor; and determining the adjustment result based on the target replication factor.

[0010] In an exemplary embodiment, the method further includes: obtaining an initial adjustment model of a distributed storage system and a historical first input feature vector, where the initial adjustment model includes an initial decision sub-model and an initial adjustment sub-model. The initial decision sub-model is established by a long short-term memory network and a feedforward neural network, and the initial adjustment sub-model is established by an embedding network, a multi-head attention network, and a feedforward neural network. The historical first input feature vector is determined by multiple historical metric parameters and multiple historical data segments of the distributed storage system; training the initial decision sub-model with the historical first input feature vector to obtain a target decision sub-model, and inputting the historical first input feature vector into the target decision sub-model for analysis to obtain a historical first output feature vector of the target decision sub-model, where the historical first output feature vector includes, in the distributed storage system, the historical adjustment probability value of the historical replication factor in the historical target group, and a historical initial output feature vector; in response to the historical adjustment probability value being greater than an adjustment probability threshold, generating a historical decision action for the historical replication factor, and based on the historical decision action, determining the historical initial output feature vector as the historical second input feature vector of the initial adjustment sub-model; splicing the historical first input feature vector and the historical second input feature vector to obtain a historical target input feature vector of the initial adjustment sub-model, and training the initial adjustment sub-model with the historical target input feature vector to obtain a target adjustment sub-model; determining a target adjustment model based on a target constraint function of the initial adjustment model, the target decision sub-model, and the target adjustment sub-model, where the target constraint function is used to constrain at least one historical metric parameter of the distributed storage system.

[0011] According to another embodiment of the present application, there is provided an adjustment device for a replication factor, including: a first acquisition module, configured to acquire multiple data segments stored in a distributed storage system; a second acquisition module, configured to perform clustering processing on the multiple data segments to obtain a clustering result of the data segments, where the clustering result is used to represent successfully aggregating the multiple data segments into multiple target groups; a generation module, configured to generate a first input feature vector of a target adjustment model corresponding to the distributed storage system based on the data segments in the multiple target groups and multiple metric parameters of the distributed storage system, where the metric parameters are used to represent the performance and / or operating state of the distributed storage system; a third acquisition module, configured to input the first input feature vector into the target adjustment model to analyze and adjust an initial replication factor of the target group to obtain an adjustment result of the initial replication factor, where the target adjustment model is established by a historical first input feature vector, and the initial replication factor is used to represent the replication times of the data segments of the target group in the distributed storage system.

[0012] According to another embodiment of the present application, there is also provided a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above method embodiments when running.

[0013] According to another embodiment of the present application, there is also provided an electronic device including a memory and a processor, where a computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any of the above method embodiments.

[0014] According to another embodiment of the present application, there is also provided a computer program product including a computer program, and the computer program implements the steps in any of the above method embodiments when executed by a processor.

[0015] Through the present application, first obtain a plurality of data segments stored in a distributed storage system, and then perform clustering processing on the obtained plurality of data segments to obtain a clustering result for characterizing the successful aggregation of the plurality of data segments into a plurality of target groups. Then, according to the data segments in the plurality of target groups and a plurality of metric parameters of the distributed storage system, generate a first input feature vector corresponding to the distributed storage system for a target adjustment model. Finally, the first input feature vector can be input into the target adjustment model to analyze and adjust the initial replication factor of the target group, so as to achieve the purpose of obtaining an adjustment result of the initial replication factor. Since it is considered that after clustering the plurality of data segments to obtain a plurality of target groups, a first input feature vector is generated according to the data segments in the plurality of target groups and a plurality of metric parameters of the distributed storage system, so that the first input feature vector can be input into the target adjustment model to analyze and adjust the initial replication factor of the target group, obtain a replication factor suitable for the above target group, and according to the obtained replication factor, in the distributed storage system, perform data replication on the data segments in the target group, so as to achieve that when it is detected that a data segment in a certain node is lost, etc., the data segment can be obtained from other nodes in time, thereby solving the technical problem of low reliability of the distributed storage system and achieving the technical effect of improving the reliability of the distributed storage system. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 is a hardware structure block diagram of a server device for a method of adjusting a replication factor according to an embodiment of the present application;

[0017] Figure 2 is a flowchart of a method of adjusting a replication factor according to an embodiment of the present application;

[0018] Figure 3 is a flowchart of a method of adjusting a replication factor of data according to an embodiment of the present application;

[0019] Figure 4 It is a schematic diagram of a dual - model structure according to an embodiment of the present application;

[0020] Figure 5 It is a structural block diagram of an adjustment device for a replication factor according to an embodiment of the present application. Detailed implementation manners

[0021] In the following, embodiments of the present application will be described in detail with reference to the accompanying drawings and in conjunction with the embodiments.

[0022] It should be noted that the terms "first", "second", etc. in the specification, claims and the above - mentioned drawings of the present application are used to distinguish similar objects, and do not necessarily need to describe a specific order or sequence.

[0023] The method embodiments provided in the embodiments of the present application can be executed in a server device or a similar computing device. Taking running on a server device as an example, Figure 1 It is a hardware structural block diagram of a server device for an adjustment method of a replication factor according to an embodiment of the present application. As Figure 1 shown, the server device may include one or more ( Figure 1 only one is shown in the figure) processors 102 (the processors 102 may include, but are not limited to, processing devices such as a micro - processor MCU or a programmable logic device FPGA) and a memory 104 for storing data. Among them, the above - mentioned server device may further include a transmission device 106 for communication functions and an input / output device 108. Those of ordinary skill in the art can understand that Figure 1 the structure shown is only schematic, and it does not limit the structure of the above - mentioned server device. For example, the server device may further include more or fewer components than Figure 1 shown in the figure, or have a different configuration from Figure 1 shown in the figure.

[0024] The memory 104 can be used to store computer programs. For example, software programs and modules of application software, such as the computer program corresponding to the adjustment method of the replication factor in the embodiments of the present application. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, implements the above - mentioned method. The memory 104 may include a high - speed random access memory, and may also include a non - volatile memory, such as one or more magnetic storage devices, flash memories, or other non - volatile solid - state memories. In some instances, the memory 104 may further include a memory remotely set relative to the processor 102, and these remote memories can be connected to the server device through a network. Examples of the above - mentioned network include, but are not limited to, the Internet, an enterprise intranet, a local area network, a mobile communication network, and their combinations.

[0025] The transmission device 106 is used to receive or send data via a network. Specific examples of the above-mentioned network may include a wireless network provided by a communication provider of a server device. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, abbreviated as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a Radio Frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0026] In this embodiment, a method for adjusting a replication factor is provided. Figure 2 It is a flowchart of a method for adjusting a replication factor according to an embodiment of the present application, as Figure 2 shown, and the process includes the following steps:

[0027] Step S202, obtain a plurality of data segments stored in a distributed storage system.

[0028] In step S202 of the embodiment of the present application, an access interface provided by the distributed storage system can be used to initiate a data reading request to the distributed storage system so as to obtain a plurality of data segments stored in the distributed storage system.

[0029] Optionally, the distributed storage system is a computer system that stores data on multiple nodes and is used to achieve efficient management and access of data in a network environment, and can be abbreviated as a system. The data segments may include attribute information such as data size, data access frequency, and data importance.

[0030] It should be noted that this is only a preferred implementation manner for obtaining a plurality of data segments stored in the distributed storage system, and the process and method for obtaining a plurality of data segments stored in the distributed storage system are not specifically limited.

[0031] Step S204, perform clustering processing on the plurality of data segments to obtain a clustering result of the data segments.

[0032] In step S204 of the embodiment of the present application, the obtained plurality of data segments are clustered by using an aggregation algorithm (K-means algorithm) so as to obtain a clustering result of the data segments, wherein the clustering result is used to represent that a plurality of data segments are successfully aggregated into a plurality of target groups. Among them, the attribute information of the target group includes: average size, average access frequency, average importance, and the number of data segments. The clustering result can be called a data aggregation result.

[0033] Optionally, the K-means algorithm is used to cluster the data segments in the system. Each data segment contains attribute information such as data size, data access frequency, and data importance. Through the clustering algorithm, a large number of independent data segments are grouped into a small number of data groups, significantly reducing the amount of data to be processed while retaining important statistical features. For example, 10,000 data segments can be aggregated into 10 groups, and each group contains attribute information such as average size, average access frequency, average importance, and the number of data segments. The features corresponding to the above attribute information are called group-level features.

[0034] It should be noted that this is only a preferred implementation manner for obtaining the clustering result of the data segments, and the process and method for obtaining the clustering result of the data segments are not specifically limited. As long as the process and method for clustering multiple data segments to obtain the clustering result of the data segments are within the protection scope of this application, they will not be listed here.

[0035] Step S206: Generate a first input feature vector of the target adjustment model corresponding to the distributed storage system based on the data segments in the multiple target groups and the multiple metric parameters of the distributed storage system.

[0036] In step S206 of the embodiment of this application, after obtaining the multiple target groups, a first input feature vector of the target adjustment model corresponding to the distributed storage system is generated according to the data segments in the multiple target groups and the multiple metric parameters of the distributed storage system, where the metric parameters are used to characterize the performance and / or operating state of the distributed storage system and can be abbreviated as metrics.

[0037] Optionally, a monitoring tool is used to continuously monitor and collect various metrics of the distributed storage system, such as storage utilization rate, average replication factor, system load, request latency, system throughput, proportion of hot data, read-write ratio, node failure rate, data recovery time, cross-region traffic, storage space, and bandwidth cost. The above metrics can comprehensively reflect the operating state and performance of the system, and the features corresponding to the above metrics can be called system-level features.

[0038] It can be understood that this is only a preferred implementation manner for obtaining the first input feature vector of the target adjustment model, and the process and method for obtaining the first input feature vector of the target adjustment model are not specifically limited. As long as the process and method for generating a first input feature vector of the target adjustment model corresponding to the distributed storage system based on the data segments in the multiple target groups and the multiple metric parameters of the distributed storage system are within the protection scope of this application, they will not be listed here.

[0039] Step S208: Input the first input feature vector into the target adjustment model to analyze and adjust the initial replication factor of the target group, and obtain the adjustment result of the initial replication factor.

[0040] In step S208 of the embodiment of the present application, the obtained first input feature vector is input into the target adjustment model to realize the analysis and adjustment of the initial replication factor of the target group, so as to achieve the purpose of obtaining the adjustment result of the initial replication factor.

[0041] Optionally, the target adjustment model is used to be established through historical first input feature vectors. The initial replication factor is used to characterize the replication times of data segments of the target group in the distributed storage system, and can be represented by a numerical value. For example, 1, 3, etc. When the replication factor is 3, it means that each piece of data has 3 copies in the system.

[0042] It should be noted that this is only a preferred implementation manner for obtaining the adjustment result of the initial replication factor, and the process and method for obtaining the adjustment result of the initial replication factor are not specifically limited. As long as the first input feature vector is input into the target adjustment model to analyze and adjust the initial replication factor of the target group and obtain the adjustment result of the initial replication factor, the process and method are within the protection scope of the present application and will not be elaborated here.

[0043] Through the above steps, first obtain multiple data segments stored in the distributed storage system, and then perform clustering processing on the obtained multiple data segments to obtain a clustering result for characterizing the successful aggregation of multiple data segments into multiple target groups. Then, according to the data segments in the multiple target groups and multiple index parameters of the distributed storage system, generate the first input feature vector of the target adjustment model corresponding to the distributed storage system. Finally, the first input feature vector can be input into the target adjustment model to analyze and adjust the initial replication factor of the target group, so as to achieve the purpose of obtaining the adjustment result of the initial replication factor. Since it is considered that after clustering multiple data segments to obtain multiple target groups, according to the data segments in the multiple target groups and multiple index parameters of the distributed storage system, generate the first input feature vector, so as to input the first input feature vector into the target adjustment model to realize the analysis and adjustment of the initial replication factor of the target group, obtain a replication factor suitable for the above target group, and according to the obtained replication factor, in the distributed storage system, perform data replication on the data segments in the target group, so as to realize that when it is detected that a data segment in a certain node is lost, etc., the data segment can be obtained from other nodes in time, thus solving the technical problem of low reliability of the distributed storage system and achieving the technical effect of improving the reliability of the distributed storage system.

[0044] As an alternative embodiment, based on data segments in multiple target groups and multiple metric parameters of a distributed storage system, generating a first input feature vector corresponding to the distributed storage system's target adjustment model includes: respectively performing feature extraction on data segments in multiple target groups to obtain first feature extraction vectors of multiple target groups; respectively performing feature extraction on multiple metric parameters to obtain second feature extraction vectors of multiple metric parameters; and generating a first input feature vector based on the first feature extraction vectors and the second feature extraction vectors.

[0045] In this embodiment, after obtaining multiple target groups and multiple metric parameters, feature extraction can be respectively performed on data segments in multiple target groups to obtain first feature extraction vectors of multiple target groups. Furthermore, feature extraction needs to be respectively performed on multiple metric parameters to obtain second feature extraction vectors of multiple metric parameters. Based on the first feature extraction vectors and the second feature extraction vectors obtained above, the purpose of generating a first input feature vector can be achieved.

[0046] Optionally, the first feature extraction vector can be referred to as a group-level feature vector. The second feature extraction vector can be referred to as a system-level feature vector. The first input feature vector can be a feature vector containing 52 elements. It should be noted that only an example of the dimension of the first input feature vector is given here, and the dimension of the first input feature vector is not specifically limited.

[0047] For example, based on the system-level feature vector and the group-level feature vector, a first input feature vector is constructed. This vector contains the collected system-level features and the group-level features (average size, access frequency, importance, and number of data segments) of each data group collected, so as to finally generate a feature vector containing 52 elements (12 system-level features + 10 groups * 4 group-level features), improving the comprehensiveness and accuracy of characterizing the performance of the distributed storage system and achieving the effect of accurately analyzing the first input feature vector.

[0048] As an alternative embodiment, inputting the first input feature vector into the target adjustment model to analyze and adjust the initial replication factor of the target group to obtain an adjustment result of the initial replication factor includes: inputting the first input feature vector into the target decision sub-model of the target adjustment model for analysis to obtain a first output feature vector of the target decision sub-model, where the first output feature vector includes an adjustment probability value of the initial replication factor and an initial output feature vector, and the target decision sub-model is used to be established through historical first input feature vectors; in response to the adjustment probability value being greater than an adjustment probability threshold, generating a decision action for the initial replication factor, where the decision action is used to represent a decision to adjust the initial replication factor; and determining the adjustment result based on the decision action, the first input feature vector, and the initial output feature vector.

[0049] In this embodiment, the obtained first input feature vector can be input into the target decision sub-model of the target adjustment model for analysis, so as to obtain the first output feature vector of the target decision sub-model. The first output feature vector can include the adjustment probability value of the initial replication factor and the initial output feature vector. By comparing the adjustment probability value with the adjustment probability threshold, if the adjustment probability value is greater than the adjustment probability threshold, a decision action for the initial replication factor can be generated. Then, based on the decision action, the first input feature vector, and the initial output feature vector, the adjustment result of the initial replication factor can be determined.

[0050] Optionally, the adjustment probability threshold can be a preset threshold, for example, 0.5. The target decision sub-model can be represented by decision-model. The initial output feature vector can be represented by the feature vector v. The first output feature vector can be a data feature vector with a dimension of (1, 41). Among them, the output value of the first data is between 0 and 1, which is used to represent the probability of whether the replication factor needs to be adjusted. If this output value is greater than the preset threshold of 0.5, it is considered that adjustment is required; if this output value is less than or equal to the preset threshold of 0.5, it is considered that the replication factor does not need to be adjusted. The values of the remaining 40 dimensions are called the feature vector v and are used as part of the input to the target adjustment sub-model of the target adjustment model, so as to accurately determine whether the replication factor needs to be adjusted at this time and achieve the effect of accurately determining the adjustment result.

[0051] As an optional embodiment, based on the decision action, the first input feature vector, and the initial output feature vector, determining the adjustment result includes: based on the decision action, determining the initial output feature vector as the second input feature vector of the target adjustment sub-model of the target adjustment model; performing a splicing process on the first input feature vector and the second input feature vector to obtain the target input feature vector of the target adjustment sub-model; inputting the target input feature vector into the target adjustment sub-model to adjust the initial replication factor to obtain the adjustment result, where the target adjustment sub-model is used to be established through historical target input feature vectors.

[0052] In this embodiment, according to the decision action used to characterize the adjustment of the initial replication factor, the initial output feature vector can be determined as the second input feature vector of the target adjustment sub-model of the target adjustment model. Then, the first input feature vector and the second input feature vector are spliced to obtain the target input feature vector of the target adjustment sub-model. The obtained target input feature vector is input into the target adjustment sub-model to adjust the initial replication factor to achieve the purpose of obtaining the adjustment result. Among them, the target adjustment sub-model can be represented by adjust-model.

[0053] For example, if the decision-model determines that the initial replication factor needs to be adjusted, the feature vector v and the output of the decision-model are concatenated and then input into the adjust-model. The adjust-model adopts a neural network structure with a multi-head attention mechanism, which can effectively extract the important features of the input features, enrich the performance features of the distributed storage system while retaining the important feature information, achieving the technical effect of effectively adjusting the initial replication factor.

[0054] As an alternative embodiment, the target input feature vector is input into the target adjustment sub-model to adjust the initial replication factor and obtain an adjustment result, including: inputting the target input feature vector into the target adjustment sub-model to obtain a second output feature vector, where the second output feature vector includes the adjustment value of the replication factor; based on the adjustment value, adjusting the initial replication factor to obtain an adjustment result.

[0055] In this embodiment, after obtaining the target input feature vector, the target input feature vector can be input into the target adjustment sub-model to obtain the adjustment value of the replication factor in the second output feature vector. Then, based on the obtained adjustment value, the initial replication factor can be adjusted to obtain an adjustment result.

[0056] Optionally, the adjustment value is used to represent the adjustment ratio of the replication factor for each data group, and its range is between -1 and 1, which also represents the degree of reduction or increase of the replication factor. The dimension of the second output feature vector is [n, 10], where n represents the number of all nodes in the distributed storage system, and 10 represents the adjustment value of the replication factor for each data group in each node. For example, 0.1, 0.5, etc.

[0057] As an alternative embodiment, based on the adjustment value, adjusting the initial replication factor to obtain an adjustment result, including: updating the initial replication factor based on the adjustment value to obtain a target replication factor corresponding to the initial replication factor; determining the adjustment result based on the target replication factor.

[0058] In this embodiment, after obtaining the adjustment value, the initial replication factor can be updated to obtain a target replication factor. Then, based on the obtained target replication factor, the adjustment result of the initial replication factor can be determined. Among them, the initial replication factor can be the current replication factor.

[0059] Optionally, the preset base value and the adjustment value are added to obtain a target sum value, where the preset base value can be a preset value, such as 1, etc.; the target sum value and the initial replication factor are multiplied to obtain a target replication factor.

[0060] For example, if the current replication factor of a certain set of data is 2 and the adjustment value is 0.5, the new replication factor will be 3, and the calculation method is 2*(1 + 0.5) = 3. At this time, the system will perform actual data replication or deletion operations accordingly, so as to select appropriate replication factors for different data segments according to the current performance of the distributed storage system, enabling each data segment to adapt to the dynamically changing load of the distributed storage system, in order to achieve the effect of improving the space utilization rate of the distributed storage system.

[0061] As an alternative embodiment, the method further includes: obtaining an initial adjustment model of the distributed storage system and a historical first input feature vector, where the initial adjustment model includes an initial decision sub-model and an initial adjustment sub-model. The initial decision sub-model is used to be established through a long short-term memory network and a feedforward neural network, and the initial adjustment sub-model is used to be established through an embedding network, a multi-head attention network and a feedforward neural network. The historical first input feature vector is used to be determined by multiple historical metric parameters and multiple historical data segments of the distributed storage system; using the historical first input feature vector to train the initial decision sub-model to obtain a target decision sub-model, and inputting the historical first input feature vector into the target decision sub-model for analysis to obtain a historical first output feature vector of the target decision sub-model, where the historical first output feature vector includes, in the distributed storage system, the historical adjustment probability value of the historical replication factor in the historical target group and a historical initial output feature vector; in response to the historical adjustment probability value being greater than an adjustment probability threshold, generating a historical decision action for the historical replication factor, and based on the historical decision action, determining the historical first input feature vector as the historical second input feature vector of the initial adjustment sub-model; concatenating the historical first input feature vector and the historical second input feature vector to obtain a historical target input feature vector of the initial adjustment sub-model, and using the historical target input feature vector to train the initial adjustment sub-model to obtain a target adjustment sub-model; determining a target adjustment model based on the target constraint function of the initial adjustment model, the target decision sub-model and the target adjustment sub-model, where the target constraint function is used to constrain at least one historical metric parameter of the distributed storage system.

[0062] In this embodiment, an initial adjustment model of the distributed storage system and historical first input feature vectors can be obtained first. The initial adjustment model includes an initial decision sub-model and an initial adjustment sub-model. The initial decision sub-model is used to be established through a long short-term memory network and a feed-forward neural network. The initial adjustment sub-model is used to be established through an embedding network, a multi-head attention network, and a feed-forward neural network. Furthermore, the historical first input feature vectors can be used to train the initial decision sub-model to obtain a target decision sub-model. And the historical first input feature vectors are input into the target decision sub-model for analysis to obtain the historical first output feature vectors of the target decision sub-model. At this time, the historical first output feature vectors include the historical adjustment probability values of the historical replication factors in the historical target group, and the historical initial output feature vectors. The historical adjustment probability values at this time are compared with the adjustment probability threshold. When the historical adjustment probability value is greater than the adjustment probability threshold, the historical decision actions of the historical replication factors can be generated, that is, the historical replication factors in the historical group at this time need to be adjusted. Further, according to the historical decision actions obtained above, the historical initial output feature vectors can be determined as the historical second input feature vectors of the initial adjustment sub-model. Then, the obtained historical first input feature vectors and historical second input feature vectors are concatenated to obtain the historical target input feature vectors of the initial adjustment sub-model. And the historical target input feature vectors are used to train the initial adjustment sub-model to obtain a target adjustment sub-model. Finally, according to the target constraint function of the initial adjustment model, the target decision sub-model, and the target adjustment sub-model, a target adjustment model can be determined. Among them, the target adjustment model can be called a dual model.

[0063] Optionally, after obtaining the target decision sub-model and the target adjustment sub-model, the target decision sub-model and the target adjustment sub-model are iteratively trained using the target constraint function to obtain a trained model, that is, the target adjustment model. Among them, the target constraint function can be a reward function, and can be composed of a target throughput rate, a target storage utilization rate, and a target cost rate, and can be represented by R.

[0064] Optionally, obtain the historical throughput, current throughput, historical storage utilization value, current storage utilization value, historical total cost, and current total cost of the distributed storage system; determine the target throughput rate based on the historical throughput and the current throughput; determine the target storage utilization rate based on the historical storage utilization value and the current storage utilization value; determine the target cost rate based on the historical total cost and the current total cost; determine the target constraint function based on the target throughput rate, the target storage utilization rate, and the target cost rate.

[0065] Furthermore, subtract the historical throughput from the current throughput to obtain a first difference, and determine the ratio of the first difference to the historical throughput as the target throughput rate. Here, the current throughput can be referred to as the new throughput, the historical throughput can be referred to as the old throughput, and the target throughput rate can be represented by ΔPerformance; subtract the historical storage utilization value from the current storage utilization value to obtain a second difference, and determine the ratio of the second difference to the historical storage utilization value as the target storage utilization rate. Here, the current storage utilization value can be referred to as the new storage utilization value, the historical storage utilization value can be referred to as the old storage utilization value, and the target storage utilization rate can be represented by ΔResourceUtilization; subtract the historical total cost from the current total cost to obtain a third difference, and determine the ratio of the third difference to the historical total cost as the target cost rate. Here, the current total cost can be referred to as the new total cost, the historical total cost can be referred to as the old total cost, and the target cost rate can be represented by ΔCost.

[0066] For example, the decision-model and adjust-model are continuously optimized using the Reinforcement Learning (RL) method. The purpose of using this model is to optimize the performance of the distributed storage system to improve its availability. Therefore, when defining the reward function, factors such as system performance improvement, resource utilization optimization, and cost reduction need to be considered. Then, the reward function can be determined by the following formula:

[0067] R = w 1 ΔPerformance + w 2 ΔResourceUtilizattion - w 3 ΔCost

[0068] From the above formula, it can be seen that ΔPerformance = (new throughput - old throughput) / old throughput, ΔResourceUtilization = (new storage utilization value - old storage utilization value) / old storage utilization value, ΔCost = (new total cost - old total cost) / old total cost, w 1 , w 2 , w 3 all represent weight coefficients and satisfy the sum of 1. It should be noted that a preset framework (actor-critic framework) is used during training. Here, the actor in the preset framework is used to represent the combination of the decision-model and adjust-model, and the critic in the preset framework is a network specifically used to train the actor. Using the critic can evaluate the impact of the two models after outputting actions and executing them.

[0069] In addition, when it comes to the training of two models, the decision - model and the adjust - model need to be trained separately. However, both are guided by the temporal difference error value (TD error value) to update the model gradients. The difference is that part of the output of the decision - model is applied in the adjust - model. Therefore, each time there is an update, the adjust - model is updated once, the decision - model is updated twice, and it is updated once by itself, that is, it is updated once along with the adjust - model. During training, experience replay is used in the buffer to store historical decisions and their results. These experiences are regularly used to update the model parameters, and a noise mechanism is used to enhance the training effect and increase the model's robustness.

[0070] It can be understood that the model needs to be iteratively looped. That is, after completing one round of adjustment, the system continues to collect new state information and prepares for the next round of decision - making. This process continues to ensure that the system can continuously adapt to environmental changes. After continuous training, the model converges and can be formally deployed in the distributed storage system. When in use, at regular intervals, the system - level features and group - level features of each node in the entire system are input into the decision - model, and then it is determined whether to modify the replication factor. If necessary, the adjust - model is used to determine the size and nodes of the factor that needs to be modified in order to perform the replication factor modification operation.

[0071] In the embodiment of this application, first, multiple data segments stored in the distributed storage system are obtained. Then, clustering processing is performed on the obtained multiple data segments to obtain a clustering result for characterizing the successful aggregation of multiple data segments into multiple target groups. Then, according to the data segments in the multiple target groups and multiple metric parameters of the distributed storage system, a first input feature vector corresponding to the target adjustment model of the distributed storage system is generated. Finally, the first input feature vector can be input into the target adjustment model to analyze and adjust the initial replication factor of the target group, so as to achieve the purpose of obtaining the adjustment result of the initial replication factor. Since it is considered that after clustering multiple data segments to obtain multiple target groups, according to the data segments in the multiple target groups and multiple metric parameters of the distributed storage system, a first input feature vector is generated, so that the first input feature vector is input into the target adjustment model to realize the analysis and adjustment of the initial replication factor of the target group, and a replication factor suitable for the above - mentioned target group is obtained. According to the obtained replication factor, data replication is performed on the data segments in the target group in the distributed storage system, so that when it is detected that a data segment in a certain node is lost or other situations occur, the data segment can be obtained from other nodes in time, thus solving the technical problem of low reliability of the distributed storage system and achieving the technical effect of improving the reliability of the distributed storage system.

[0072] Through the description of the above embodiments, those skilled in the art can clearly understand that the method according to the above embodiments can be implemented by means of software plus a necessary general hardware platform. Of course, it can also be implemented by hardware, but in many cases, the former is a better implementation method. Based on such an understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes several instructions for causing a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods of the various embodiments of the present application.

[0073] The following uses preferred embodiments to illustrate the technical solutions of the embodiments of the present invention.

[0074] In a distributed storage system, data replication is a key technology to improve data availability and system performance, and can be achieved by setting a replication factor. Among them, data replication refers to saving copies of the same data on multiple storage nodes, and the replication factor indicates the number of times each piece of data is replicated. For example, when the replication factor is 3, it means that each piece of data has 3 copies in the system.

[0075] Currently, by adopting a static replication strategy, that is, setting the same fixed replication factor for all data, data replication in a distributed storage system is realized. However, the above method depends on manually set rules, lacks flexibility and adaptability, and is difficult to adapt to the dynamically changing load of the distributed storage system, resulting in the technical problem of low reliability of the distributed storage system.

[0076] To solve the above problems, a method for dynamically adjusting the replication factor based on reinforcement learning is proposed. This method designs a dual-model structure to realize the intelligent and automatic adjustment of the replication factor in a distributed storage system. The method includes a decision model and a dynamic adjustment model for modifying the replication factor. Based on these two models, an adaptive reinforcement learning framework is constructed. This framework dynamically determines the replication factor in the distributed storage system by analyzing system information such as the storage utilization rate of the system and the load conditions of storage nodes. And the model can balance data availability, system performance, and storage overhead, and output the optimal replication factor configuration under different system loads and data characteristics, thereby solving the technical problem of low reliability of the distributed storage system and achieving the technical effect of improving the reliability of the distributed storage system.

[0077] Figure 3 is a flowchart of a method for adjusting the replication factor of data according to an embodiment of the present application. As Figure 3 shown, the method mainly includes the following steps:

[0078] Step S301, obtain the system information of the distributed storage system.

[0079] In this embodiment, the system information of the distributed storage system can be obtained. A monitoring tool is used to continuously monitor and collect various metrics of the distributed storage system, such as storage utilization rate, average replication factor, system load, request latency, system throughput, proportion of hot data, read-write ratio, node failure rate, data recovery time, cross-region traffic, storage space, and bandwidth cost. The above metrics can comprehensively reflect the operating status and performance of the system, and the characteristics corresponding to the above metrics can be called system-level characteristics.

[0080] Step S302, input the system information into the decision model.

[0081] In this embodiment, multiple data segments stored in the distributed storage system are obtained, and the system information and multiple data segments can be input into the decision model, where the decision model can be represented by decision-model.

[0082] Optionally, the K-means algorithm is used to cluster the data segments in the system. Each data segment contains attribute information such as data size, data access frequency, and data importance. Through the clustering algorithm, a large number of independent data segments are grouped into a small number of data groups, which greatly reduces the amount of data to be processed while retaining important statistical features. For example, 10,000 data segments can be aggregated into 10 groups, and each group contains attribute information such as average size, average access frequency, average importance, and the number of data segments. The characteristics corresponding to the above attribute information are called group-level characteristics.

[0083] Furthermore, based on the system-level feature vector and the group-level feature vector, a first input feature vector is constructed. This vector contains the collected system-level features and the group-level features (average size, access frequency, importance, and the number of data segments) of each data group collected, so as to finally generate a feature vector containing 52 elements (12 system-level features + 10 groups * 4 group-level features).

[0084] Optionally, the implementation process of the decision model decision-model: input the feature vector constructed by each node into decision-model. Considering that the number of nodes in each distributed storage system is different, decision-model adopts the LSTM and multi-layer perceptron structures. At the initial stage of the model, the LSTM network is used to process each node to increase the availability of the model. After feature extraction, the feature vector is input into the multi-layer perceptron for processing, and finally a data with a dimension of (1,41) is output.

[0085] Step S303, whether to adjust the replication factor.

[0086] In this embodiment, it is necessary to determine whether to adjust the replication factor. If so, step S304 is executed; if not, the process ends directly.

[0087] Optionally, according to the above steps, data with a dimension of (1, 41) can be output. Among them, the value of the first data is between 0 and 1, which is used to represent the probability of whether the replication factor needs to be adjusted. If the output value is greater than a preset threshold (such as 0.5), it is considered that adjustment is required, and the values of the remaining 40 dimensions are called the feature vector v and are used as part of the input to the adjust-model.

[0088] Step S304: Generate a decision-making action.

[0089] In this embodiment, if the replication factor needs to be adjusted, a decision-making action is generated.

[0090] Step S305: Input the decision-making action and system information into the adjustment model.

[0091] In this embodiment, the decision-making action and system information obtained in the above steps can be input into the adjustment model, and the adjustment model can be represented by adjust-model.

[0092] Step S306: Determine the adjustment ratio of the replication factor.

[0093] In this embodiment, the adjustment ratio of the replication factor can be determined through the adjustment model. The implementation process of the adjustment model adjust-model is as follows: If the decision-model determines that adjustment is required, the feature vector v and the input vector of the decision-model are concatenated and then input into the adjust-model. The adjust-model adopts a neural network structure with a multi-head attention mechanism, and the dimension of the output data is [n, 10], where n is used to represent the number of all nodes in the distributed storage system, and 10 represents the adjustment value of the replication factor of each data group in each node. The range of this adjustment value is between -1 and 1. For example, 0.1, 0.5, etc.

[0094] Optionally, according to the output of the adjust-model, update the replication factor of the data segments in each data group. For example, if the current replication factor of a certain group of data is 2 and the adjustment value is 0.5, the new replication factor will be 3, and the calculation method is 2*(1 + 0.5) = 3. At this time, the system will perform actual data replication or deletion operations accordingly.

[0095] In this embodiment, if the model needs to be trained, the training process can be as follows: The reinforcement learning (RL) method is used to continuously optimize the decision-model and the adjust-model. The purpose of using this model is to optimize the performance of the distributed storage system so as to improve the availability of the distributed storage system. Therefore, when defining the reward function, factors such as system performance improvement, resource utilization optimization, and cost reduction need to be considered. Then, the reward function can be determined by the following formula:

[0096] R = w 1 ΔPerformance + w 2 ΔResourceUtilization - w 3 ΔCost

[0097] As can be seen from the above formula, ΔPerformance = (new throughput - old throughput) / old throughput, ΔResourceUtilization = (new storage utilization value - old storage utilization value) / old storage utilization value, ΔCost = (new total cost - old total cost) / old total cost, and w 1 , w 2 , w 3 all represent weight coefficients and satisfy the sum of 1. It should be noted that a preset framework (actor-critic framework) is used during training. Among them, the actor in the preset framework is used to represent the combination of the decision-model and the adjust-model, and the critic in the preset framework is a network specifically used to train the actor. The critic can evaluate the impact of the two models after outputting actions and executing them.

[0098] In addition, regarding the training of the two models, the decision-model and the adjust-model need to be trained separately, but both use the time difference error value (TD error value) as a guide to update the model gradient. The difference is that part of the output of the decision-model is applied in the adjust-model. Therefore, each time the adjust-model is updated once, the decision-model is updated twice, and it is updated once itself, that is, it is updated once with the adjust-model. During training, experience replay is used in the buffer to store historical decisions and their results, and these experiences are used to update the model parameters regularly, and a noise mechanism is used to improve the training effect and increase the robustness of the model.

[0099] It can be understood that the model needs to be iterated cyclically. That is, after a round of adjustment is completed, the system continues to collect new state information and prepares for the next round of decision-making. This process continues to ensure that the system can continuously adapt to environmental changes. After continuous training, the model converges and can be formally deployed in the distributed storage system. When in use, every once in a while, the system-level features and group-level features of each node in the entire system are input into the decision-model, and then it is judged whether to modify the replication factor. If necessary, the adjust-model is used to determine the size and nodes of the factor that needs to be modified in order to perform the replication factor modification operation.

[0100] In this embodiment, considering that after clustering multiple data segments to obtain multiple target groups, a first input feature vector is generated according to the data segments in the multiple target groups and multiple metric parameters of the distributed storage system, so as to input the first input feature vector into the target adjustment model to analyze and adjust the initial replication factor of the target group, obtain a replication factor suitable for the above target group, and according to the obtained replication factor, in the distributed storage system, data replication is performed on the data segments in the target group, so that when it is detected that a data segment in a certain node is lost or the like, the data segment can be obtained from other nodes in time, thus solving the technical problem of low reliability of the distributed storage system and achieving the technical effect of improving the reliability of the distributed storage system.

[0101] In the embodiment of the present application, a method for dynamically adjusting the replication factor based on reinforcement learning is proposed. This method adopts a dual-model structure, including a decision model and an adjustment model. The specific solution is as follows: First, the system collects comprehensive state information, including metrics in multiple dimensions such as storage utilization, system load, and request latency. Then, the K-means algorithm is used to aggregate the data segments to reduce the decision space. Next, an input vector is constructed through feature engineering, including system-level features and data group features. Among them, the decision model is used to judge whether to adjust the replication factor based on these features. If necessary, the adjustment model determines the specific adjustment plan for each data group. Finally, the system updates the replication factor of the data segments according to the model output, thus solving the technical problem of low reliability of the distributed storage system and achieving the technical effect of improving the reliability of the distributed storage system.

[0102] Figure 4 is a schematic diagram of a dual-model structure according to an embodiment of the present application, as Figure 4As shown in the figure, the dual-model structure includes: a decision-making model and an adjustment model. Among them, the decision-making model is composed of a long short-term memory network (Long Short-Term Memory Network, abbreviated as LSTM) and a feedforward neural network (MLP network), and the adjustment model can be composed of an embedding network (Embedding network), a multi-head attention network, and a feedforward neural network.

[0103] Optionally, in the decision-making model, system information can be obtained first, and then the system information is input into the LSTM network to obtain a first output feature. The first output feature is input into the feedforward neural network to generate a decision-making action and a feature vector v. In the adjustment model, the output data in the decision-making model, that is, the feature vector v and the obtained system information, are input into the embedding network to obtain a second output feature. The second output feature is input into the multi-head attention network to obtain a third output feature. The third output feature is input into the feedforward neural network to obtain a replication factor.

[0104] In the embodiment of the present application, a method for dynamically adjusting the replication factor based on reinforcement learning is proposed. This method adopts a dual-model structure, including a decision-making model and an adjustment model. The specific solution is as follows: First, the system collects comprehensive state information, including indicators in multiple dimensions such as storage utilization rate, system load, and request delay. Then, the K-means algorithm is used to aggregate data segments to reduce the decision-making space. Next, an input vector is constructed through feature engineering, including system-level features and data group features. Among them, the decision-making model is used to determine whether the replication factor needs to be adjusted based on these features. If so, the adjustment model determines the specific adjustment plan for each data group. Finally, the system updates the replication factor of the data segment according to the model output, thereby solving the technical problem of low reliability of the distributed storage system and achieving the technical effect of improving the reliability of the distributed storage system.

[0105] In this embodiment, an adjustment device for the replication factor is also provided. This device is used to implement the above-mentioned embodiments and preferred implementation manners, and those that have been described will not be repeated. As used hereinafter, the term "module" can be a combination of software and / or hardware that can implement a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, implementation in hardware, or a combination of software and hardware is also possible and contemplated.

[0106] Figure 5 is a structural block diagram of an adjustment device for the replication factor according to an embodiment of the present application. As Figure 5 shown, the device includes: a first acquisition module 501, a second acquisition module 502, a generation module 503, and a third acquisition module 504.

[0107] The first acquisition module 501 is configured to acquire a plurality of data segments stored in a distributed storage system.

[0108] The second acquisition module 502 is configured to perform clustering processing on the plurality of data segments to obtain a clustering result of the data segments, where the clustering result is used to characterize that the plurality of data segments are successfully aggregated into a plurality of target groups.

[0109] The generation module 503 is configured to generate a first input feature vector of a target adjustment model corresponding to the distributed storage system based on the data segments in the plurality of target groups and a plurality of metric parameters of the distributed storage system, where the metric parameters are used to characterize the performance and / or operating state of the distributed storage system.

[0110] The third acquisition module 504 is configured to input the first input feature vector into the target adjustment model to analyze and adjust the initial replication factor of the target group to obtain an adjustment result of the initial replication factor, where the target adjustment model is established by using historical first input feature vectors, and the initial replication factor is used to characterize the replication times of the data segments of the target group in the distributed storage system.

[0111] Optionally, the generation module 503 is further configured to respectively perform feature extraction on the data segments in the plurality of target groups to obtain first feature extraction vectors of the plurality of target groups; respectively perform feature extraction on the plurality of metric parameters to obtain second feature extraction vectors of the plurality of metric parameters; and generate a first input feature vector based on the first feature extraction vectors and the second feature extraction vectors.

[0112] Optionally, the third acquisition module 504 is further configured to input the first input feature vector into a target decision sub-model of the target adjustment model for analysis to obtain a first output feature vector of the target decision sub-model, where the first output feature vector includes an adjustment probability value of the initial replication factor and an initial output feature vector, and the target decision sub-model is established by using historical first input feature vectors; in response to the adjustment probability value being greater than an adjustment probability threshold, generate a decision action for the initial replication factor, where the decision action is used to characterize a decision to adjust the initial replication factor; and determine an adjustment result based on the decision action, the first input feature vector, and the initial output feature vector.

[0113] Optionally, the third acquisition module 504 is further configured to determine the initial output feature vector as a second input feature vector of a target adjustment sub-model of the target adjustment model based on the decision action; perform splicing processing on the first input feature vector and the second input feature vector to obtain a target input feature vector of the target adjustment sub-model; and input the target input feature vector into the target adjustment sub-model to adjust the initial replication factor to obtain an adjustment result, where the target adjustment sub-model is established by using historical target input feature vectors.

[0114] Optionally, the third acquisition module 504 is further configured to input the target input feature vector into the target adjustment sub-model to obtain a second output feature vector, where the second output feature vector includes an adjustment value of the replication factor; based on the adjustment value, adjust the initial replication factor to obtain an adjustment result.

[0115] Optionally, the third acquisition module 504 is further configured to update the initial replication factor based on the adjustment value to obtain a target replication factor corresponding to the initial replication factor; based on the target replication factor, determine the adjustment result.

[0116] Optionally, the apparatus further includes: a fourth acquisition module, configured to acquire an initial adjustment model of the distributed storage system and a historical first input feature vector, where the initial adjustment model includes an initial decision sub-model and an initial adjustment sub-model, the initial decision sub-model is established by a long short-term memory network and a feed-forward neural network, the initial adjustment sub-model is established by an embedding network, a multi-head attention network and a feed-forward neural network, and the historical first input feature vector is determined by multiple historical metric parameters and multiple historical data segments of the distributed storage system; a fifth acquisition module, configured to use the historical first input feature vector to train the initial decision sub-model to obtain a target decision sub-model, and input the historical first input feature vector into the target decision sub-model for analysis to obtain a historical first output feature vector of the target decision sub-model, where the historical first output feature vector includes a historical adjustment probability value of the historical replication factor in the historical target group in the distributed storage system, and a historical initial output feature vector; a first determination module, configured to generate a historical decision action of the historical replication factor in response to the historical adjustment probability value being greater than an adjustment probability threshold, and based on the historical decision action, determine the historical initial output feature vector as a historical second input feature vector of the initial adjustment sub-model; a sixth acquisition module, configured to splice the historical first input feature vector and the historical second input feature vector to obtain a historical target input feature vector of the initial adjustment sub-model, and use the historical target input feature vector to train the initial adjustment sub-model to obtain a target adjustment sub-model; a second determination module, configured to determine a target adjustment model based on a target constraint function of the initial adjustment model, the target decision sub-model and the target adjustment sub-model, where the target constraint function is used to constrain at least one historical metric parameter of the distributed storage system.

[0117] In this embodiment, a plurality of data segments stored in a distributed storage system are obtained through a first obtaining module; the plurality of data segments are clustered through a second obtaining module to obtain a clustering result of the data segments, where the clustering result is used to represent that the plurality of data segments are successfully aggregated into a plurality of target groups; a first input feature vector of a target adjustment model corresponding to the distributed storage system is generated through a generating module based on the data segments in the plurality of target groups and a plurality of metric parameters of the distributed storage system, where the metric parameters are used to represent the performance and / or operating state of the distributed storage system; the first input feature vector is input into the target adjustment model through a third obtaining module to analyze and adjust an initial replication factor of the target group, and an adjustment result of the initial replication factor is obtained, where the target adjustment model is used to be established through historical first input feature vectors, and the initial replication factor is used to represent the number of replications of the data segments of the target group in the distributed storage system, thereby solving the technical problem of low reliability of the distributed storage system and achieving the technical effect of improving the reliability of the distributed storage system.

[0118] It should be noted that the above-mentioned various modules can be implemented by software or hardware. For the latter, it can be implemented in the following ways, but not limited to: the above-mentioned modules are all located in the same processor; or, the above-mentioned various modules are separately located in different processors in any combination form.

[0119] An embodiment of the present application also provides a computer-readable storage medium, in which a computer program is stored, where the computer program is configured to execute the steps in any one of the above method embodiments when running.

[0120] In an exemplary embodiment, the above computer-readable storage medium may include, but is not limited to: a USB flash drive, a read-only memory (ROM for short), a random access memory (RAM for short), a mobile hard disk, a magnetic disk, or an optical disc and other various media that can store computer programs.

[0121] An embodiment of the present application also provides an electronic device, including a memory and a processor, where a computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0122] In an exemplary embodiment, the above electronic device may further include a transmission device and an input / output device, where the transmission device is connected to the above processor, and the input / output device is connected to the above processor.

[0123] Embodiments of the present application also provide a computer program product. The computer program product includes a computer program, and when the computer program is executed by a processor, the steps in any of the above method embodiments are implemented.

[0124] Embodiments of the present application also provide another computer program product, including a non-volatile computer-readable storage medium. The non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in any of the above method embodiments are implemented.

[0125] Embodiments of the present application also provide a computer program. The computer program includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the steps in any of the above method embodiments.

[0126] Specific examples in this embodiment may refer to the examples described in the above embodiments and exemplary embodiments, and will not be repeated here.

[0127] Obviously, those skilled in the art should understand that the above modules or steps of the present application can be implemented by a general-purpose computing device. They can be concentrated on a single computing device or distributed on a network composed of multiple computing devices. They can be implemented by program codes executable by the computing device. Thus, they can be stored in a storage device and executed by the computing device. And in some cases, the steps shown or described can be executed in a different order from here, or they can be separately made into individual integrated circuit modules, or multiple modules or steps among them can be made into a single integrated circuit module to implement. In this way, the present application is not limited to any specific combination of hardware and software.

[0128] The above are only the preferred embodiments of the present application and are not used to limit the present application. For those skilled in the art, the present application can have various changes and modifications. Any modification, equivalent replacement, improvement, etc. made within the principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method for adjusting a replication factor, characterized in that: include: Obtain multiple data fragments stored in a distributed storage system; Performing clustering processing on the plurality of data segments to obtain clustering results of the data segments, wherein the clustering results are used to indicate that the plurality of data segments are successfully aggregated into a plurality of target groups; Generate a first input feature vector of a target adjustment model corresponding to the distributed storage system based on the data segments in the plurality of target groups and a plurality of indicator parameters of the distributed storage system, wherein the indicator parameters are used to characterize the performance and / or operating status of the distributed storage system; The first input feature vector is input into the target adjustment model, and the initial replication factor of the target group is analyzed and adjusted to obtain an adjustment result of the initial replication factor, wherein the target adjustment model is used to be established through the historical first input feature vector, and the initial replication factor is used to characterize the number of replications of the data fragments of the target group in the distributed storage system.

2. The method according to claim 1, characterized in that The step of generating a first input feature vector of a target adjustment model corresponding to the distributed storage system based on the data segments in the plurality of target groups and the plurality of indicator parameters of the distributed storage system comprises: Performing feature extraction on the data segments in the plurality of target groups respectively to obtain first feature extraction vectors of the plurality of target groups; Performing the feature extraction on the plurality of indicator parameters respectively to obtain second feature extraction vectors of the plurality of indicator parameters; The first input feature vector is generated based on the first feature extraction vector and the second feature extraction vector.

3. The method according to claim 1, characterized in that The step of inputting the first input feature vector into the target adjustment model, analyzing and adjusting the initial replication factor of the target group, and obtaining an adjustment result of the initial replication factor includes: Inputting the first input feature vector into the target decision sub-model of the target adjustment model for analysis to obtain a first output feature vector of the target decision sub-model, wherein the first output feature vector includes an adjusted probability value of the initial replication factor and an initial output feature vector, and the target decision sub-model is used to be established by using the historical first input feature vector; In response to the adjustment probability value being greater than the adjustment probability threshold, generating a decision action for the initial replication factor, wherein the decision action is used to represent a decision to adjust the initial replication factor; The adjustment result is determined based on the decision action, the first input feature vector and the initial output feature vector.

4. The method according to claim 3, characterized in that The determining the adjustment result based on the decision action, the first input feature vector and the initial output feature vector comprises: Based on the decision action, determining the initial output feature vector as a second input feature vector of a target adjustment sub-model of the target adjustment model; Concatenating the first input feature vector and the second input feature vector to obtain a target input feature vector of the target adjustment sub-model; The target input feature vector is input into the target adjustment sub-model, and the initial replication factor is adjusted to obtain the adjustment result, wherein the target adjustment sub-model is used to be established through the historical target input feature vector.

5. The method according to claim 4, characterized in that The step of inputting the target input feature vector into the target adjustment sub-model, adjusting the initial replication factor, and obtaining the adjustment result includes: Inputting the target input feature vector into the target adjustment sub-model to obtain a second output feature vector, wherein the second output feature vector includes an adjustment value of the replication factor; Based on the adjustment value, the initial replication factor is adjusted to obtain the adjustment result.

6. The method according to claim 5, characterized in that The adjusting the initial replication factor based on the adjustment value to obtain the adjustment result includes: Based on the adjustment value, the initial replication factor is updated to obtain a target replication factor corresponding to the initial replication factor; Based on the target replication factor, the adjustment result is determined.

7. The method according to any one of claims 1 to 6, characterized in that The method further comprises: Acquire an initial adjustment model of the distributed storage system and the first historical input feature vector, wherein the initial adjustment model includes an initial decision sub-model and an initial adjustment sub-model, the initial decision sub-model is used to be established through a long short-term memory network and a feedforward neural network, the initial adjustment sub-model is used to be established through an embedding network, a multi-head attention network and the feedforward neural network, and the first historical input feature vector is used to be determined through multiple historical indicator parameters and multiple historical data fragments of the distributed storage system; The initial decision sub-model is trained by using the first historical input feature vector to obtain a target decision sub-model, and the first historical input feature vector is input into the target decision sub-model for analysis to obtain a first historical output feature vector of the target decision sub-model, wherein the first historical output feature vector includes a historical adjustment probability value of a historical replication factor in a historical target group in the distributed storage system, and a historical initial output feature vector; in response to the historical adjustment probability value being greater than the adjustment probability threshold, a historical decision action of the historical replication factor is generated, and based on the historical decision action, the historical initial output feature vector is determined as a second historical input feature vector of the initial adjustment sub-model; The historical first input feature vector and the historical second input feature vector are concatenated to obtain the historical target input feature vector of the initial adjustment sub-model, and the historical target input feature vector is used to train the initial adjustment sub-model to obtain the target adjustment sub-model; The target adjustment model is determined based on the target constraint function of the initial adjustment model, the target decision sub-model and the target adjustment sub-model, wherein the target constraint function is used to constrain at least one of the historical indicator parameters of the distributed storage system.

8. A device for adjusting a replication factor, characterized in that: include: A first acquisition module, used to acquire multiple data fragments stored in the distributed storage system; A second acquisition module is used to perform clustering processing on the multiple data segments to obtain clustering results of the data segments, wherein the clustering results are used to indicate that the multiple data segments are successfully aggregated into multiple target groups; A generating module, configured to generate a first input feature vector of a target adjustment model corresponding to the distributed storage system based on the data segments in the plurality of target groups and a plurality of indicator parameters of the distributed storage system, wherein the indicator parameters are used to characterize the performance and / or operating status of the distributed storage system; The third acquisition module is used to input the first input feature vector into the target adjustment model, analyze and adjust the initial replication factor of the target group, and obtain the adjustment result of the initial replication factor, wherein the target adjustment model is used to be established through the historical first input feature vector, and the initial replication factor is used to characterize the number of replications of the data fragments of the target group in the distributed storage system.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program implements the steps of the method described in any one of claims 1 to 7 when executed by a processor.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method described in any one of claims 1 to 7 are implemented.

Images