Method, device, electronic equipment, medium and product for optimizing storage system

Abstract

The application discloses an optimization method and device of a storage system, electronic equipment, medium and product, and relates to the technical field of data management. Since the global hotness prediction model can comprehensively analyze the meta feature vector, the storage strategy matched with the current data access behavior can be predicted, so that the accuracy of the storage system optimization is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data management, and particularly relates to an optimization method and device of a storage system, electronic equipment, a medium and a product. BACKGROUND

[0002] In the application scenarios of data, such as cloud storage platforms and distributed computing, a large amount of data can be safely and long-term stored through a storage system, and data reading is performed when the data needs to be used. Since the storage system stores a large amount of data, the read-write performance of the storage system is challenged.

[0003] In the related art, a fixed threshold is used for decision-making to adjust the storage strategy of the storage system, so as to optimize the performance of the storage system. However, the error of this method is large, resulting in low optimization accuracy of the storage system. SUMMARY

[0004] The present application provides an optimization method and device of a storage system, electronic equipment, a medium and a product to at least solve the problem of low optimization accuracy of the storage system in the related art.

[0005] The present application provides an optimization method of a storage system, comprising:

[0006] receiving a data access request, wherein the data access request comprises data access information and a target data block of a target storage system;

[0007] performing feature extraction processing on the data access information according to the data access request to obtain a meta-feature vector;

[0008] inputting the meta-feature vector into a pre-trained global hotness prediction model of the target storage system to obtain a predicted access frequency of the target data block;

[0009] performing confidence calculation through a meta-learner according to the meta-feature vector and the predicted access frequency to obtain a confidence calculation result, and determining a storage strategy according to the confidence calculation result;

[0010] adjusting the storage location of the target data block according to the storage strategy to perform optimization processing on the storage system.

[0011] The present application also provides an optimization device of a storage system, comprising:

[0012] a receiving module configured to receive a data access request, wherein the data access request comprises data access information and a target data block of a target storage system;

[0013] an extraction module configured to perform feature extraction processing on the data access information according to the data access request to obtain a meta-feature vector;

[0014] a prediction module, configured to input the meta-feature vector into a pre-trained global hotness prediction model of the target storage system to obtain a predicted access frequency of the target data block;

[0015] a calculation module, configured to perform confidence calculation through a meta-learner according to the meta-feature vector and the predicted access frequency to obtain a confidence calculation result, and determine a storage strategy according to the confidence calculation result;

[0016] an optimization module, configured to adjust a storage location of the target data block according to the storage strategy to perform optimization processing on the storage system.

[0017] The application further provides an electronic device, including a memory configured to store a computer program, and a processor configured to execute the computer program to implement the steps of the optimization method of the storage system.

[0018] The application further provides a non-volatile computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the steps of the optimization method of the storage system.

[0019] The application further provides a computer program product, which includes a computer program, and the computer program is executed by a processor to implement the steps of the optimization method of the storage system.

[0020] According to the application, since the global hotness prediction model can comprehensively analyze the meta-feature vector, the storage strategy matched with the current data access behavior can be predicted, so that the accuracy of the optimization of the storage system is improved. BRIEF DESCRIPTION OF DRAWINGS

[0021] In order to more clearly illustrate the embodiments of the application, the drawings needed in the embodiments will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the application, and other drawings can be obtained by those skilled in the art without creative labor.

[0022] Figure 1 An application scenario diagram of the optimization method of the storage system provided by the embodiments of the application;

[0023] Figure 2 A flowchart of the optimization method of the storage system provided by the embodiments of the application;

[0024] Figure 3 A flowchart of another optimization method of the storage system provided by the embodiments of the application;

[0025] Figure 4 A schematic diagram of generating a global hotness prediction model is provided for the embodiments of the present application;

[0026] Figure 5 A schematic diagram of determining a storage strategy is provided for the embodiments of the present application;

[0027] Figure 6 A schematic diagram of a probe injection strategy is provided for the embodiments of the present application;

[0028] Figure 7 A structural schematic diagram of an optimization device of a storage system is provided for the embodiments of the present application;

[0029] Figure 8 A structural schematic diagram of another optimization device of a storage system is provided for the embodiments of the present application;

[0030] Figure 9 A structural schematic diagram of an electronic device is provided for the embodiments of the present application. DETAILED DESCRIPTION

[0031] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0032] It should be noted that, in the description of the present application, the terms “comprise”, “contain” or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device comprising a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or device. The terms “first”, “second” and the like in the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence.

[0033] With the rapid development of cloud storage platforms, distributed computing and other scenarios, the amount of data is growing explosively, and higher requirements are put forward for the I / O performance and intelligent level of the storage system. Modern storage systems usually adopt a multi-layer storage architecture, that is, a high-speed storage medium such as dynamic random access memory (DRAM) is used as a cache layer, and a low-speed but larger capacity persistent storage medium such as a solid-state drive (SSD) or a mechanical hard disk is used as a main storage. The core optimization goal of the storage system is to keep the hot data that is most likely to be accessed in the cache as much as possible to reduce access delay and improve overall system performance.

[0034] In practical applications, the commonly used storage strategies mainly include two categories: Write-Through, data is written to cache and main storage at the same time. The Write-Through strategy ensures the data consistency of cache and main storage, but the write delay is high, and if the data is not accessed after being written, it will occupy limited cache space.

[0035] Write-Around, data is written directly to the main storage when writing. The Write-Around strategy avoids the occupation of cache space by cold data, but when the data is read later, it will cause high read delay due to cache miss.

[0036] In order to enable those skilled in the art to better understand the present application, the present application will be further described in detail below in combination with the drawings and specific embodiments.

[0037] In combination with the specific application scenario architecture on which the execution of the optimization method of the storage system depends, the specific application scenario architecture is described here. Referring to Figure 1 , Figure 1 The application scenario diagram of the optimization of the storage system. The storage system stores a plurality of data, and the user end accesses the data in the storage system through the storage system to realize the reading and / or writing of the data in the storage system.

[0038] In the related art, the storage strategy is determined only according to a fixed threshold, such as a simple historical access frequency, least recently used and its variants algorithm. However, these methods have significant limitations. The related art passively reacts based on the historical access frequency, which cannot adapt to the current application scenario and has a cold start problem. Therefore, the storage strategy determined in the related art has a large error, resulting in the problem of low accuracy of the storage strategy.

[0039] Figure 2 The flowchart of the optimization method of the storage system provided by the embodiment of the present application is shown in Figure 2As shown, embodiments of the present application provide an optimization method of a storage system, which is described in detail as follows:

[0040] S201, receiving a data access request, the data access request comprising data access information and a target data block of a target storage system.

[0041] For example, the data access information is related information of a user's access behavior to the storage system.

[0042] Optionally, the data access request is an I / O request under a protocol such as Small Computer System Interface (SCSI) or Non-Volatile Memory Express (NVMe).

[0043] Optionally, the data access information comprises at least one of the following: a source of the data access request (which can be an address of a virtual machine or a container), a type of the data access request (read / write), a logical volume access mode (which can be a random or continuous write flag), or a data size, etc.

[0044] For example, the target data block is a data block of the storage system to which the data access request is specifically directed. By specifying the target data block, the optimization granularity can be optimized, thereby improving the refinement and feasibility of the optimization operation.

[0045] In combination with a scenario example, when the data access request is received, an optimization of the storage system is triggered to be performed to determine a matched storage strategy for data storage, thereby optimizing the performance of the storage system.

[0046] S202, performing feature extraction processing on the data access information according to the data access request to obtain a meta-feature vector.

[0047] For example, features related to data are extracted from the data access information, and the features are combined to obtain the meta-feature vector.

[0048] Optionally, the extracted features can comprise a data producer identifier (such as a virtual machine identifier or a container group), a logical volume access mode identifier (such as a random or continuous write flag), and semantic information of data access (generated by analyzing a SCSI command context). The data producer identifier is used to identify the data source and reflect the complexity of the producer behavior. The logical volume access mode identifier is used to describe the continuity of the data access behavior. The semantic information of data access is used to reflect the semantic information of the data to determine the association between the data.

[0049] Optionally, the meta-feature vector is generated by the following formula:

[0050]

[0051] wherein, represents a metadata vector, represents a data producer identity, represents an access pattern identity, represents semantic information of data access.

[0052] Optionally, the Shannon entropy is calculated by the data producer identity to measure the complexity of the producer behavior. The continuity of data access behavior is determined by calculating the LBA distribution of the logical volume access pattern. The correlation between data is determined by calculating the frequency of occurrence of relevant data of the semantic information of data access in the transaction log.

[0053] Illustratively, the data access information is converted into a meta-feature vector that can be understood and recognized by a machine, so as to make accurate prediction through a model.

[0054] S203, input the meta-feature vector into a pre-trained global hotness prediction model of the target storage system to obtain a predicted access frequency of the target data block.

[0055] Illustratively, the global hotness prediction model is obtained by fusing a plurality of local hotness prediction models corresponding to a plurality of logical volumes. By fusing a plurality of models, the effect of federated learning can be achieved to solve the problems of single-point data sparsity and poor generalization ability.

[0056] Illustratively, the predicted access frequency is used to quantify the hotness of the data in the target data block. The higher the predicted access frequency, the closer the data in the target data block to hot data.

[0057] Illustratively, pre-training is a strategy for training a deep learning model, the core of which is to use a large-scale data set to preliminarily train the model, so that the model learns a general feature representation. This process is similar to the basic learning stage before humans learn new knowledge, through extensive reading, observation and experience accumulation.

[0058] In combination with a scene example, the predicted access frequency generated by the global hotness prediction model can comprehensively identify the data access information compared to a single fixed threshold, thereby improving the accuracy of identification.

[0059] Optionally, the calculation formula of the predicted access frequency is:

[0060]

[0061] wherein, represents the predicted access frequency, represents an activation function, a multi-layer perception model is represented, a global hotness prediction model is represented, a meta-feature vector is represented.

[0062] S204, according to the meta-feature vector and the predicted access frequency, confidence calculation is performed through the meta-learner to obtain a confidence calculation result, and a storage strategy is determined according to the confidence calculation result.

[0063] Exemplarily, the confidence calculation result is used to evaluate the reliability of the predicted access frequency. Thus, a secondary decision is made, and the predicted access frequency is verified according to the secondary decision.

[0064] Exemplarily, an uncertainty evaluation and dynamic strategy selection mechanism is introduced, which avoids the risk brought by blind decision-making, thereby improving the accuracy of storage system optimization.

[0065] Exemplarily, according to the confidence calculation result, a matching storage strategy is determined.

[0066] Optionally, the formula of confidence calculation is:

[0067]

[0068] wherein, represents the confidence calculation result, i represents the index of the target data block, represents the historical access frequency of the target data block, represents the predicted access frequency of the target data block, represents the feature vector closest to the current meta-feature vector, which is found from the feature library.

[0069] S205, according to the storage strategy, the storage location of the target data block is adjusted to optimize the storage system.

[0070] Exemplarily, the storage strategy is a strategy that can improve the performance of the storage system. The storage strategy is converted into a specific operation, thereby improving the performance of the storage system.

[0071] In combination with a scene example, if it is predicted that the data written in the target data block is hot data, a transparent write strategy is executed. If it is predicted that the data written in the target data block is cold data, a bypass write strategy is executed. Thus, after each data access, the cache hit rate can be accurately improved, cache pollution can be avoided, and hard disk write amplification can be reduced, thereby continuously optimizing the storage system.

[0072] The optimization method of the storage system provided in the embodiments of the present application comprises the following steps: receiving a data access request, wherein the data access request comprises data access information and a target data block of a target storage system; performing feature extraction processing on the data access information according to the data access request to obtain a meta feature vector; inputting the meta feature vector into a pre-trained global hotness prediction model of the target storage system to obtain a predicted access frequency of the target data block; performing confidence calculation through a meta learner according to the meta feature vector and the predicted access frequency to obtain a confidence calculation result, and determining a storage strategy according to the confidence calculation result; and adjusting the storage location of the target data block according to the storage strategy to perform optimization processing on the storage system. In the above solution, since the global hotness prediction model can comprehensively analyze the meta feature vector, the storage strategy that matches the current data access behavior can be predicted, thereby improving the accuracy of the optimization of the storage system.

[0073] On the basis of any one of the above embodiments, the following will be described in combination with Figure 3 The detailed process of the optimization of the storage system is described.

[0074] Figure 3 Another flowchart of the optimization method of the storage system provided in the embodiments of the present application is shown in FIG. 4. As shown in FIG. 4, the method comprises the following steps. Figure 3

[0075] S301, receiving a data access request, wherein the data access request comprises data access information and a target data block of a target storage system.

[0076] It should be noted that the execution process of S301 is described in S201, which will not be described here.

[0077] S302, determining context information of the data access information.

[0078] For example, the context information is related to the data access request, and the context information is used to enrich the content of the data access information and supplement the data access information.

[0079] Optionally, the context information can supplement the missing content in the data access information, for example, the source of the data access request or the user process.

[0080] S303, performing analysis processing on the data access information and the context information to obtain target features, wherein the target features comprise at least one of the following: data producer identifier, access mode identifier, and semantic information of data access.

[0081] For example, compared with only performing analysis processing on the data access information, performing analysis processing on the data access information and the context information can obtain more sufficient and multi-dimensional target features, thereby accurately reflecting the features of the data access request.

[0082] ​In an implementation, the data access information and the context information are input into a graph neural network for feature extraction, and target features are obtained.

[0083] In an example, the data access information and the context information are input into the graph neural network, and the graph neural network extracts target features corresponding to specified feature types from the data access information and the context information.

[0084] In an example, the graph neural network learns to extract highly abstract features from low-level and raw information during training. The graph neural network can capture the essential properties of the information, and thus accurately generate the target features.

[0085] In the implementation, the accuracy of the target features is improved by using the graph neural network, and thus the accuracy of the storage system optimization is improved.

[0086] In S304, a meta-feature vector is generated according to the target features.

[0087] In an implementation, the meta-feature vector can be generated by determining a corresponding sliding time window, obtaining historical access requests according to the sliding time window and the data access information, determining an access interval feature and a periodicity model feature of the target data block according to the historical access requests, and generating the meta-feature vector according to the target features, the access interval feature, and the periodicity model feature.

[0088] In an example, the sliding time window is a dynamically adjustable time range, which can be adaptively adjusted.

[0089] Optionally, a matching sliding time window is dynamically determined according to the execution time and the accuracy requirement, so as to balance the consumption of resources and the accuracy of optimization.

[0090] In an example, the historical access requests related to the data access information are determined within the time range corresponding to the sliding time window, and the historical access requests are used as references.

[0091] In an example, the data access information reflects a specific application scenario corresponding to the data access request, and the historical access requests associated with the application scenario are selected to obtain the access rule of the target data block, so as to accurately generate the meta-feature vector.

[0092] In an example, the access interval feature represents the distribution rule between adjacent historical access requests, for example, dense distribution or interval distribution.

[0093] In an example, the periodicity model feature represents the periodicity rule of the historical access requests, for example, once every 2 minutes or once every 10 minutes.

[0094] Exemplarily, on the basis of the target feature, the access interval feature and the periodicity model feature are combined to form a meta feature vector.

[0095] In combination with a scene example, a data block that is usually accessed sparsely suddenly starts high-frequency access, and the decision method in the related art through a fixed threshold has a problem of slow reaction. The present application can accurately generate a meta feature vector by obtaining the law of historical access requests.

[0096] In this feasible implementation manner, multiple information is comprehensively considered to accurately generate a meta feature vector, thereby improving the accuracy of storage system optimization.

[0097] S305, inputting the meta feature vector into a pre-trained global hotness prediction model of the target storage system to obtain a predicted access frequency of the target data block.

[0098] A feasible implementation manner, the generation method of the global hotness prediction model can include: determining system information of the target storage system; determining a plurality of logical volumes for storing data of the target storage system from the system information; determining a plurality of local hotness prediction models corresponding to the plurality of logical volumes and a plurality of data access frequencies, wherein the plurality of local hotness prediction models are obtained by model training on a plurality of training data extracted from transaction logs of the storage system according to a lightweight feature extraction mechanism; determining a plurality of hot data weights corresponding to the plurality of logical volumes according to the plurality of data access frequencies; adding differential noise to the plurality of local hotness prediction models respectively to obtain a plurality of processing prediction models; and performing aggregation processing on the plurality of processing prediction models according to the plurality of hot data weights and a difference constraint threshold to obtain the global hotness prediction model, wherein the difference constraint threshold is used to constrain the difference between each processing prediction model and the global hotness prediction model.

[0099] Exemplarily, the system information is overall information of the target storage system, from which the plurality of logical volumes are identified. The logical volume is an independent unit managed by the storage system, which can be understood as a different disk partition or a dedicated storage space of a tenant.

[0100] Exemplarily, each logical volume locally maintains a local hotness prediction model (denoted as ), which is trained based on the access history of the logical volume itself, and therefore it best understands the access law of its own data.

[0101] Exemplarily, the data access frequencies of the logical volumes are counted, which is an index measuring the activity and importance of the logical volume.

[0102] Exemplarily, a hot data weight (denoted as ) is calculated for each logical volume according to the plurality of access frequencies. The higher the access frequency of the logical volume, the higher its weight The larger the hotness is, the more important the logical volume is in the next model aggregation.

[0103] Hereinafter, the embodiments of the present application will be described in detail with reference to the accompanying drawings. Figure 4 The global hotness prediction model is generated.

[0104] Figure 4 The global hotness prediction model is generated. Figure 4 As shown in the schematic diagram, the global hotness prediction model is obtained by aggregating the plurality of local hotness prediction models maintained by the plurality of logical volumes.

[0105] For example, the global hotness prediction model obtained by aggregating the plurality of processing prediction models through federated learning can accurately reflect the respective weights of the plurality of local hotness prediction models.

[0106] Optionally, the formula of model aggregation is:

[0107]

[0108]

[0109] wherein, represents the global hotness prediction model, k represents the index of the logical volume, K represents the total number of logical volumes, represents the hot data weight, represents the local hotness prediction model, represents the hotness of the kth logical volume, represents the hotness of the ith logical volume.

[0110] For example, differential noise processing can protect the privacy of the original data of each logical volume.

[0111] Optionally, the local hotness prediction model is processed by differential noise before uploading the local hotness prediction model, so as to avoid that the original data is deduced from the model parameters by external equipment. By adding differential noise, a plurality of processing prediction models are obtained, and the global hotness prediction model is obtained by aggregating the processing prediction models instead of the local hotness prediction model.

[0112] Optionally, the differential noise processing can be performed by Gaussian noise or Laplace noise.

[0113] In this feasible implementation manner, the hot data weight is introduced for weighted aggregation, so that the global hotness prediction model is more inclined to learn the most active and most frequently accessed data pattern, thereby accurately reflecting the hot data rule and further improving the accuracy of storage system optimization.

[0114] Optionally, the aggregation processing can be performed by a method comprising: determining a difference constraint threshold; dynamically adjusting the plurality of processing prediction models according to the difference constraint threshold, and performing aggregation processing on the adjusted processing prediction models to obtain a global hotness prediction model, the difference constraint threshold being used to constrain the difference between each processing prediction model and the global hotness prediction model.

[0115] For example, the difference constraint threshold defines the difference between each processing prediction model and the intermediate global model during the aggregation process. If the difference of the processing prediction model is less than the difference constraint threshold, the processing prediction model is directly aggregated. If the difference of the processing prediction model is greater than or equal to the difference constraint threshold, it indicates that there may be abnormal data in the processing prediction model, and the processing prediction model is adjusted and then aggregated.

[0116] Optionally, the adjustment is performed by cropping or scaling, etc.

[0117] Optionally, the difference is described by calculating the Euclidean distance between the model parameters.

[0118] In this feasible implementation manner, the model divergence problem caused by Non-IID data and outliers can be avoided by the difference constraint, thereby improving the accuracy of the storage system optimization.

[0119] Optionally, the plurality of local hotness prediction models can be determined by a method comprising: determining a data type of the training data; extracting a plurality of training data of a plurality of logical volumes from a transaction log of a storage system according to a lightweight feature extraction mechanism and the data type; and performing model training with the plurality of training data as training samples to obtain the plurality of local hotness prediction models.

[0120] For example, the transaction log is a highly structured data file maintained by the storage system to record all I / O operation metadata.

[0121] For example, the data type determines which type of training data needs to be collected, and the data is extracted from the transaction log according to the field corresponding to the data type to obtain the plurality of training data.

[0122] Optionally, the data type includes but is not limited to at least one of the following: logical block address, access timestamp, operation type, request source identifier, or return status, etc.

[0123] In this feasible implementation manner, the training data is extracted from the transaction log by the lightweight feature extraction mechanism, which can avoid full-scan of the full data, thereby improving the efficiency of model training.

[0124] S306, according to the meta feature vector and the predicted access frequency, performing confidence calculation through the meta learner to obtain a confidence calculation result, and determining a storage strategy according to the confidence calculation result.

[0125] In an implementation, the meta learner is a multi-layer perception model, and the confidence calculation can be performed by inputting the meta feature vector and the predicted access frequency into the meta learner to enable the meta learner to perform confidence calculation through an activation function to obtain the confidence calculation result.

[0126] For example, the multi-layer perception model is a feedforward artificial neural network, which includes an input layer, one or more hidden layers, and an output layer.

[0127] For example, the multi-layer perception model performs weighted summation and nonlinear transformation on input features, and gradually learns a complex and nonlinear relationship between the input features and the confidence of the prediction result. In the output layer, a specific activation function is used to map the final linear calculation result to a probability value between 0 and 1. The probability value is determined as the confidence calculation result.

[0128] Optionally, the activation function is Sigmoid or Softmax.

[0129] In this implementation, the multi-layer perception model is used to capture complex nonlinear relationships, thereby improving the accuracy of the confidence calculation result and the accuracy of the optimization of the storage system.

[0130] In an implementation, the optimization method of the storage system can further include determining an update parameter according to the confidence calculation result, and updating the meta learner according to the update parameter.

[0131] For example, the update parameter is a feedback parameter for the calculation result output by the meta learner.

[0132] Optionally, the behavior of the storage system after performing the confidence calculation result is monitored to determine whether the confidence calculation result is accurate or whether the performance of the storage system is optimized, thereby generating the update parameter.

[0133] Optionally, the parameters of the meta learner are fine-tuned according to the update parameter, thereby realizing the update processing.

[0134] In this implementation, the dynamic update mechanism can gradually improve the accuracy of the meta learner, thereby improving the accuracy of the optimization of the storage system.

[0135] A feasible implementation manner can determine the storage strategy by the following method, comprising: determining the corresponding first confidence threshold and second confidence threshold according to the data access information; if the predicted access frequency is greater than the access frequency threshold, and the confidence calculation result is greater than or equal to the first confidence threshold, determining the storage strategy as the transparent write strategy, the transparent write strategy is to write the target data block into the cache and the main storage; if the confidence calculation result is less than the first confidence threshold and greater than or equal to the second confidence threshold, determining the storage strategy as the bypass write strategy, the second confidence threshold is less than the first confidence threshold, and the bypass write strategy is to write the target data block into the main storage; if the confidence calculation result is less than the second confidence threshold, determining the storage strategy as the probe injection strategy.

[0136] For example, the access frequency threshold is used to distinguish high heat and low heat. If the predicted access frequency is greater than the access frequency threshold, it means that the predicted target data block is high heat.

[0137] For example, the storage strategy is determined by a double-threshold criterion mechanism. The transparent write strategy is to write the target data block into the cache and the main storage. The bypass write strategy is to write the target data block into the main storage bypassing the cache. The probe injection strategy is a delayed decision.

[0138] In the following, combined with Figure 5 The determination of the storage strategy is described.

[0139] Figure 5 The schematic diagram for determining the storage strategy provided by the embodiments of the present application is shown. As Figure 5 shown, three intervals are divided by the first confidence threshold and the second confidence threshold, and the corresponding storage strategy is determined according to the interval in which the confidence calculation result is located.

[0140] Combined with the scene example, only the hot data with high confidence can enter the cache, and the cold data with low confidence is only stored in the main storage.

[0141] In this feasible implementation manner, the cache resources can be accurately allocated through the confidence calculation result, thereby improving the accuracy of the optimization of the storage system.

[0142] S307, adjusting the storage position of the target data block according to the storage strategy to optimize the storage system.

[0143] A feasible implementation manner, the storage strategy is the probe injection strategy; the storage position of the target data block can be adjusted by the following method, comprising: determining the new data in the target data block, and writing the new data into the training data buffer through the idle hardware accelerator; obtaining a plurality of access information for the new data; determining an updated storage strategy according to the plurality of access information and the meta-learner; and adjusting the storage position of the target data block through the updated storage strategy.

[0144] For example, the new data is data written in the target data block according to the data access request.

[0145] Next, combined with Figure 6 The probe injection strategy is described.

[0146] Figure 6 The schematic diagram of the probe injection strategy provided by the embodiment of the application is shown. As Figure 6 shown, for the target data block of the probe injection strategy, the target data block is written into the training data buffer. A monitoring process is started to monitor subsequent multiple access information of the target data block, and a secondary decision is made according to the multiple access information to determine a matched storage strategy.

[0147] Combined with a scene example, in a scene where the confidence calculation result is medium, that is, insufficient information, the final decision is postponed to after there is sufficient behavior data through the probe injection strategy.

[0148] Optionally, the hardware accelerator includes but is not limited to at least one of the following: a graphics processing unit (GPU) or a field-programmable gate array (FPGA), etc.

[0149] For example, according to the current load of the hardware accelerator, an idle hardware accelerator is determined to improve the efficiency of writing.

[0150] For example, the efficiency of writing can be improved through the accelerator, thereby improving the efficiency of optimizing the storage system.

[0151] In this feasible implementation manner, through the probe injection strategy, blind decision can be avoided, thereby reducing the probability of misjudgment to improve the accuracy of optimizing the storage system.

[0152] Through the description of the above embodiments, those skilled in the art can clearly understand that the method according to the above embodiments can be realized by means of software and necessary general hardware platforms, of course, it can also be realized by hardware, but in many cases, the former is a better implementation manner.

[0153] Figure 7 The structural schematic diagram of the optimization device of the storage system provided by the embodiment of the application is shown. As Figure 7 shown, the embodiment of the application further provides an optimization device of a storage system, and the optimization device 70 of the storage system can include a receiving module 71, an extraction module 72, a prediction module 73, a calculation module 74, and an optimization module 75.

[0154] The receiving module 71 is configured to receive a data access request, the data access request comprising data access information and a target data block of a target storage system.

[0155] The extracting module 72 is configured to perform feature extraction processing on the data access information according to the data access request to obtain a meta feature vector.

[0156] The predicting module 73 is configured to input the meta feature vector into a pre-trained global hotness prediction model of the target storage system to obtain a predicted access frequency of the target data block.

[0157] The calculating module 74 is configured to perform confidence calculation on the meta feature vector and the predicted access frequency by using a meta-learner to obtain a confidence calculation result, and determine a storage strategy according to the confidence calculation result.

[0158] The optimizing module 75 is configured to adjust a storage location of the target data block according to the storage strategy to perform optimization processing on the storage system.

[0159] Optionally, the receiving module 71 can perform S201 in the embodiment. Figure 2

[0160] Optionally, the extracting module 72 can perform S202 in the embodiment. Figure 2

[0161] Optionally, the predicting module 73 can perform S203 in the embodiment. Figure 2

[0162] Optionally, the calculating module 74 can perform S204 in the embodiment. Figure 2

[0163] Optionally, the optimizing module 75 can perform S205 in the embodiment. Figure 2

[0164] It should be noted that the optimization apparatus of the storage system shown in the embodiments of the present application can perform the technical solutions shown in the above method embodiments, and the implementation principles and beneficial effects are similar, which will not be described here in detail.

[0165] In one possible implementation, the extracting module 72 is specifically configured to:

[0166] determine context information of the data access information;

[0167] perform analysis processing on the data access information and the context information to obtain target features, the target features comprising at least one of the following: data producer identification, access mode identification, and semantic information of data access;

[0168] generate the meta feature vector according to the target features. ​​​​​

[0169] In a possible implementation, the extraction module 72 is specifically configured to:

[0170] determine a corresponding sliding time window;

[0171] obtain a historical access request according to the sliding time window and the data access information;

[0172] determine an access interval feature and a periodic model feature of the target data block according to the historical access request;

[0173] generate the meta-feature vector according to the target feature, the access interval feature, and the periodic model feature.

[0174] In a possible implementation, the extraction module 72 is specifically configured to:

[0175] input the data access information and the context information into a graph neural network for feature extraction, to obtain the target feature.

[0176] Figure 8 Another structural schematic diagram of the optimization device of the storage system provided by the embodiment of the present application is provided. In the embodiment shown in Figure 7 the basis of the embodiment shown in Figure 8 , the optimization device 70 of the storage system further includes an execution module 76, an update module 77, a judgment module 78, an adjustment module 79, and a training module 710.

[0177] The meta-learner is a multi-layer perception model; the execution module 76 is configured to:

[0178] input the meta-feature vector and the predicted access frequency into the meta-learner, so that the meta-learner performs confidence calculation through an activation function to obtain a confidence calculation result.

[0179] The update module 77 is configured to:

[0180] determine an update parameter according to the confidence calculation result;

[0181] update the meta-learner according to the update parameter.

[0182] The judgment module 78 is configured to:

[0183] determine a first confidence threshold and a second confidence threshold according to the data access information;

[0184] if the predicted access frequency is greater than the access frequency threshold, and the confidence calculation result is greater than or equal to the first confidence threshold, it is determined that the storage strategy is a transparent write strategy, and the transparent write strategy is to write the target data block into the cache and the main storage;

[0185] If the confidence calculation result is less than the first confidence threshold and greater than or equal to the second confidence threshold, the storage strategy is determined to be a write-around strategy, the second confidence threshold is less than the first confidence threshold, and the write-around strategy is to write the target data block to the main storage;

[0186] If the confidence calculation result is less than the second confidence threshold, the storage strategy is determined to be a probe injection strategy.

[0187] The storage strategy is a probe injection strategy; and the adjusting module 79 is configured to:

[0188] Determine new data in the target data block, and write the new data to the training data buffer through the idle hardware accelerator;

[0189] Obtain a plurality of access information for the new data;

[0190] Determine an updated storage strategy according to the plurality of access information and the meta-learner;

[0191] Adjust the storage location of the target data block through the updated storage strategy.

[0192] The training module 710 is configured to:

[0193] Determine system information of a target storage system;

[0194] From the system information, determine a plurality of logical volumes of the target storage system for storing data;

[0195] Determine a plurality of local hotness prediction models corresponding to the plurality of logical volumes and a plurality of data access frequencies, wherein the plurality of local hotness prediction models are obtained by model training on a plurality of training data extracted from transaction logs of the storage system according to a lightweight feature extraction mechanism;

[0196] Determine a plurality of hot data weights corresponding to the plurality of logical volumes according to the plurality of data access frequencies;

[0197] Add differential noise to the plurality of local hotness prediction models to obtain a plurality of processed prediction models;

[0198] Aggregate the plurality of processed prediction models according to the plurality of hot data weights and a difference constraint threshold to obtain a global hotness prediction model, and the difference constraint threshold is used to constrain the difference between each processed prediction model and the global hotness prediction model.

[0199] Figure 9 The structure of the electronic device provided in the present application is shown in the figure. As shown in the figure, Figure 9As shown, the electronic device 90 provided by the embodiment includes at least one processor 901 and a memory 902. Optionally, the electronic device 90 further includes a communication component 903. The processor 901, the memory 902 and the communication component 903 are connected through a bus.

[0200] In the implementation process, the at least one processor 901 executes the computer execution instructions stored in the memory 902, so that the at least one processor 901 executes the above-mentioned storage system optimization method embodiment.

[0201] The specific implementation process of the processor 901 can refer to the above-mentioned method embodiments, which have similar implementation principles and technical effects, and will not be described here in detail.

[0202] In the above-mentioned embodiments, it should be understood that the processor can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC) and the like. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor and the like. The steps of the method disclosed in the application can be directly embodied as execution completed by a hardware processor, or executed by a combination of hardware and software modules in the processor.

[0203] The memory can include a random access memory (RAM), and can also include a non-volatile memory (NVM), for example, at least one disk memory.

[0204] The bus can be an industry standard architecture (ISA) bus, a peripheral component (PCI) bus or an extended industry standard architecture (EISA) bus and the like. The bus can be divided into an address bus, a data bus, a control bus and the like. For the convenience of representation, the bus in the drawings of the application does not limit to only one bus or one type of bus.

[0205] The embodiment of the application further provides a non-volatile computer readable storage medium, and the non-volatile computer readable storage medium stores a computer program, wherein the computer program is set to execute the steps in any one of the above-mentioned storage system optimization method embodiments when running.

[0206] In an example embodiment, the non-volatile computer readable storage medium described above can include, but is not limited to, a U disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a mobile hard disk, a magnetic disk or an optical disk, and various media that can store computer programs.

[0207] Embodiments of the present application also provide a computer program product, which comprises a computer program, and the computer program, when executed by a processor, implements the steps in any of the optimization method embodiments of the storage system described above.

[0208] Embodiments of the present application also provide another computer program product, which comprises a non-volatile computer readable storage medium, and the non-volatile computer readable storage medium stores a computer program, and the computer program, when executed by a processor, implements the steps in any of the optimization method embodiments of the storage system described above.

[0209] The skilled person can further realize that the units and algorithm steps of the examples described in connection with the embodiments disclosed herein can be realized in electronic hardware, computer software or a combination of both. In order to clearly illustrate the interchangeability of hardware and software, the components and steps of the examples have been described in general terms in the above description. Whether the functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. The skilled person can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0210] The above describes in detail the optimization method, device, electronic equipment, medium and product of a storage system provided by the present application. The principles and implementation modes of the present application are described by applying specific examples in this paper, and the above description of the examples is only applicable to help understand the method and core idea of the present application. It should be noted that, for those skilled in the art, without departing from the principles of the present application, some improvements and modifications can be made to the present application, and these improvements and modifications also fall within the protection scope of the claims of the present application.

Claims

1. An optimization method of a storage system, characterized by, The method comprises: receiving a data access request, the data access request comprising data access information and a target data block of a target storage system; performing feature extraction processing on the data access information according to the data access request to obtain a meta feature vector; inputting the meta feature vector into a pre-trained global hotness prediction model of the target storage system to obtain a predicted access frequency of the target data block; performing confidence calculation on the meta feature vector and the predicted access frequency by a meta learner to obtain a confidence calculation result, and determining a storage strategy according to the confidence calculation result; adjusting the storage location of the target data block according to the storage strategy to optimize the storage system; determining a storage strategy according to the confidence calculation result, comprising: determining a corresponding first confidence threshold and a second confidence threshold according to the data access information; if the predicted access frequency is greater than an access frequency threshold, and the confidence calculation result is greater than or equal to the first confidence threshold, determining that the storage strategy is a transparent write strategy, and the transparent write strategy is to write the target data block into a cache and a main storage; if the confidence calculation result is less than the first confidence threshold and greater than or equal to the second confidence threshold, determining that the storage strategy is a bypass write strategy, the second confidence threshold is less than the first confidence threshold, and the bypass write strategy is to write the target data block into the main storage; if the confidence calculation result is less than the second confidence threshold, determining that the storage strategy is a probe injection strategy.

2. The optimization method of a storage system according to claim 1, wherein, performing feature extraction processing on the data access information according to the data access request to obtain a meta feature vector, comprising: determining context information of the data access information; performing analysis processing on the data access information and the context information to obtain target features, the target features comprising at least one of the following: data producer identifier, access mode identifier, and semantic information of data access; generating the meta feature vector according to the target features.

3. The optimization method of a storage system according to claim 2, wherein, generating the meta feature vector according to the target features, comprising: determining a corresponding sliding time window; acquiring historical access requests according to the sliding time window and the data access information; determining access interval features and periodic model features of the target data block according to the historical access requests; generating the meta feature vector according to the target features, the access interval features, and the periodic model features.

4. The optimization method of a storage system according to claim 2, wherein, performing analysis processing on the data access information and the context information to obtain target features, comprising: inputting the data access information and the context information into a graph neural network for feature extraction to obtain the target features.

5. The optimization method of a storage system according to claim 1, wherein, The meta learner is a multi-layer perception model; performing confidence calculation on the meta feature vector and the predicted access frequency by a meta learner to obtain a confidence calculation result, comprising: inputting the meta feature vector and the predicted access frequency into the meta learner to enable the meta learner to perform confidence calculation by an activation function to obtain the confidence calculation result.

6. The optimization method of a storage system according to claim 5, wherein, The method further comprises: According to the confidence calculation result, determine an update parameter; According to the update parameter, update the meta-learner.

7. The optimization method of a storage system according to claim 1, wherein, The storage strategy is a probe injection strategy. According to the storage strategy, adjust the storage location of the target data block, including: Determine the newly added data in the target data block, and write the newly added data into a training data buffer through an idle hardware accelerator; Obtain a plurality of access information for the newly added data; According to the plurality of access information and the meta-learner, determine an updated storage strategy; Adjust the storage location of the target data block through the updated storage strategy.

8. The method of optimizing a storage system according to any of claims 1-7, wherein, The method further comprises: Determine the system information of the target storage system; From the system information, determine a plurality of logical volumes of the target storage system for storing data; Determine a plurality of local hotness prediction models corresponding to the plurality of logical volumes, and a plurality of data access frequencies, wherein the plurality of local hotness prediction models are obtained by model training on a plurality of training data extracted from transaction logs of the storage system according to a lightweight feature extraction mechanism; According to the plurality of data access frequencies, determine a plurality of hot data weights corresponding to the plurality of logical volumes; Add differential noise to the plurality of local hotness prediction models respectively to obtain a plurality of processed prediction models; According to the plurality of hot data weights and a difference constraint threshold, aggregate the plurality of processed prediction models to obtain the global hotness prediction model, and the difference constraint threshold is used to constrain the difference between each processed prediction model and the global hotness prediction model.

9. An electronic device, comprising: Comprise: A memory for storing a computer program; A processor for executing the computer program to implement the steps of the optimization method of the storage system according to any one of claims 1 to 8.

Images