Data storage method and device, equipment, storage medium and program product

Abstract

The invention discloses a data storage method and device, equipment, a storage medium and a program product, which are applied to electronic equipment, and the specific technical scheme comprises the following steps: obtaining to-be-stored data; the to-be-stored data are input into the intelligent classifier, a target storage layer output by the intelligent classifier is obtained, and the target storage layer is used for storing the to-be-stored data; obtaining a preset weight of each storage node in the target storage layer, polling each storage node according to the preset weight corresponding to each storage node, and determining a target storage node; and storing the to-be-stored data in the target storage node, and generating a corresponding relationship between the target storage node and the to-be-stored data. Therefore, the storage node corresponding to the to-be-stored node is determined according to the preset weight and the intelligent classifier, and the system performance of the hierarchical storage system is improved.

Description

Technical Field

[0001] The present application belongs to the field of big data edge technology, and in particular relates to a method, apparatus, device, storage medium and program product for data storage. Background Art

[0002] Tiered storage and data management are common technologies in cloud computing and edge computing. Traditional cloud computing models utilize a centralized storage architecture, storing data in central data centers. Edge computing, on the other hand, shifts some computing and storage resources to the edge of the network to reduce latency and improve real-time performance. Tiered storage technology typically categorizes data into hot, warm, and cold data, storing it in storage media with varying performance and cost based on access frequency and importance.

[0003] However, in the process of data tiered storage, it is usually managed based on file paths, which leads to poor data storage location and thus reduces system performance. Summary of the Invention

[0004] The embodiments of the present application provide a data storage method, apparatus, device, storage medium, and program product, which can improve the system performance of a tiered storage system.

[0005] In a first aspect, an embodiment of the present application provides a data storage method, applied to an electronic device, the method comprising:

[0006] Get the data to be stored;

[0007] Inputting the data to be stored into an intelligent classifier to obtain a target storage layer output by the intelligent classifier, wherein the target storage layer is used to store the data to be stored;

[0008] Obtaining a preset weight of each storage node in the target storage layer, and polling each storage node according to the preset weight corresponding to each storage node to determine the target storage node;

[0009] The data to be stored is stored in the target storage node, and a corresponding relationship between the target storage node and the data to be stored is generated.

[0010] In a possible implementation, before inputting the data to be stored into an intelligent classifier to obtain a target storage layer output by the intelligent classifier, and using the target storage layer to store the data to be stored, the method further includes:

[0011] Inputting the training data into the time feature extraction network, content feature extraction network and access pattern feature extraction network in the neural network model respectively, obtaining the time series features extracted by the time feature extraction network, the content features extracted by the content feature extraction network and the access pattern features extracted by the access pattern feature extraction network;

[0012] The time series features, the content features, and the access pattern features are combined by the neural network model to obtain a target vector;

[0013] Inputting the target vector into the multilayer perceptron in the neural network model to obtain the prediction result;

[0014] Calculating a loss function value of the label data and the prediction result;

[0015] The loss function value is used to adjust the model parameters of the neural network model to obtain the intelligent classifier.

[0016] In a possible implementation, storing the data to be stored in the target storage node and generating a correspondence between the target storage node and the data to be stored includes:

[0017] Obtaining a data identifier of the data to be stored;

[0018] Calculate a first hash value corresponding to the data identifier according to a preset hash algorithm;

[0019] Mapping the first hash value to a preset hash ring to obtain a first hash position;

[0020] For each virtual storage node corresponding to the target storage node, calculating a second hash value of the virtual storage node according to the preset hash algorithm;

[0021] Mapping the second hash value to the preset hash ring to obtain a second hash position corresponding to each virtual storage node;

[0022] Determine, in the second hash position corresponding to each virtual storage node, a target hash position that is the same as the first hash position;

[0023] A correspondence between the data identifier, the target hash position, and the target storage node is constructed.

[0024] In a possible implementation, the method further includes:

[0025] Obtain key indicator information for each storage node;

[0026] For each storage node, determining actual load information of the storage node according to the key indicator information;

[0027] For each storage node, the number of virtual storage nodes corresponding to the storage node is adjusted according to the actual load information and a preset step size, and the virtual storage nodes are used to store data.

[0028] In a possible implementation, the method further includes:

[0029] Obtain status information and network information of each storage node;

[0030] Constructing system status representation information according to the status information and the network information;

[0031] For target data in each storage node that meets preset scheduling conditions, calculate the instantaneous reward of the target data under different actions under the system state representation information according to a preset reward function;

[0032] Scheduling the target data according to the instant reward;

[0033] When the target data is scheduled, a corresponding relationship between the target data and the storage node is generated.

[0034] In a possible implementation, the method further includes:

[0035] For each storage node, obtaining historical access data of the storage node;

[0036] Inputting the historical access data into an access pattern prediction model to obtain a predicted access pattern;

[0037] Determining a target cache algorithm corresponding to the predicted access pattern based on a preset correspondence between the access pattern and the cache algorithm;

[0038] The cache space of different types of data in the storage node is adjusted based on the target cache algorithm.

[0039] In a second aspect, an embodiment of the present application provides a data storage device, applied to an electronic device, the device comprising:

[0040] An acquisition module, used to acquire data to be stored;

[0041] a classification module, configured to input the data to be stored into an intelligent classifier, and obtain a target storage layer output by the intelligent classifier, wherein the target storage layer is used to store the data to be stored;

[0042] A polling module is used to obtain a preset weight of each storage node in the target storage layer, and poll each storage node according to the preset weight corresponding to each storage node to determine the target storage node;

[0043] The storage module is used to store the data to be stored in the target storage node and generate a corresponding relationship between the target storage node and the data to be stored.

[0044] In a third aspect, an embodiment of the present application provides a terminal device, the device comprising: a processor and a memory storing computer program instructions;

[0045] When the processor executes the computer program instructions, the data storage method according to the first aspect is implemented.

[0046] In a fourth aspect, an embodiment of the present application provides a computer storage medium, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the method for data storage as in the first aspect is implemented.

[0047] In a fifth aspect, an embodiment of the present application provides a computer program product. When the instructions in the computer program product are executed by a processor of an electronic device, the electronic device executes the data storage method as in the first aspect.

[0048] A data storage method, apparatus, device, storage medium and program product of an embodiment of the present application, after obtaining the data to be stored, uses an intelligent classifier to determine the target storage layer of the data to be stored. In this way, by distinguishing the storage layers of different data, hierarchical storage of data can be achieved to achieve effective utilization of edge computing resources. Then, within the target storage layer, according to the preset weight of each storage node in the target storage layer, polling is performed according to the preset weight of each storage node to determine the target storage node of the data to be stored. In this way, according to the preset weight, dynamic load can be achieved within the target storage layer, thereby improving the overall performance of data storage. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0050] Figure 1 This is a structural diagram of a data storage system architecture provided by an embodiment of the present application;

[0051] Figure 2 This is a flow chart of a data storage method provided by an embodiment of the present application;

[0052] Figure 3 This is a flow chart of a method for training an intelligent classifier provided in an embodiment of the present application;

[0053] Figure 4 This is an exemplary schematic diagram of an intelligent classifier workflow provided in an embodiment of the present application;

[0054] Figure 5 This is an exemplary schematic diagram of a local load balancing method provided in an embodiment of the present application;

[0055] Figure 6 This is a flow chart of a method for constructing data index information provided in an embodiment of the present application;

[0056] Figure 7 This is an exemplary schematic diagram of a data query method provided in an embodiment of the present application;

[0057] Figure 8 This is a flow chart of a data scheduling method provided in an embodiment of the present application;

[0058] Figure 9 is an exemplary schematic diagram of a data scheduling method provided in an embodiment of the present application;

[0059] Figure 10 This is a flow chart of a data caching method provided in an embodiment of the present application;

[0060] Figure 11 This is an exemplary schematic diagram of a data caching method provided in an embodiment of the present application;

[0061] Figure 12 This is an exemplary schematic diagram of a method for storing, accessing, and scheduling data provided by an embodiment of the present application;

[0062] Figure 13 is an exemplary schematic diagram of a fault detection and data recovery method provided in an embodiment of the present application;

[0063] Figure 14 This is a schematic diagram of the structure of a data storage device provided in an embodiment of the present application;

[0064] Figure 15 It is a structural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0065] The features and exemplary embodiments of various aspects of the present application will be described in detail below. In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain the present application, rather than to limit the present application. For those skilled in the art, the present application can be implemented without the need for some of these specific details. The following description of the embodiments is merely to provide a better understanding of the present application by illustrating the examples of the present application.

[0066] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising..." does not exclude the presence of additional identical elements in the process, method, article, or device comprising the element.

[0067] Current methods for allocating and scheduling data for edge computing resources have limitations. For example, they underutilize edge computing resources, failing to meet the low-latency and real-time requirements for data queries. Alternatively, after data is stored in tiers, there's a lack of data management mechanisms. Furthermore, they focus on resource and data scheduling at edge computing nodes without integrating with regional data centers or central clouds.

[0068] In order to solve the problems of the prior art, the embodiments of the present application provide a method, apparatus, device, storage medium and program product for data storage. Figure 1 Introducing the system architecture of the data storage method provided by this application, such as Figure 1 As shown, Figure 1 The system architecture of a data storage method provided in an embodiment of the present application includes three storage layers and four core components.

[0069] Core components include the Intelligent Data Classifier (IDC), Dynamic Data Scheduler (DDS), Distributed Metadata Manager (DMM), and Adaptive Cache Manager (ACM). The storage layer consists of the Edge Layer (EL), Regional Layer (RL), and Central Cloud Layer (CCL).

[0070] The edge layer is deployed on edge nodes, such as base stations or edge servers; the regional layer is deployed in regional data centers; and the central cloud layer is deployed in the central cloud. Storage nodes in all three storage layers can be used to store data.

[0071] The intelligent data classifier is used to assign different storage layers to the received data to be stored;

[0072] The dynamic data scheduler is used to schedule data based on network conditions, node storage capacity, and data access patterns;

[0073] The distributed metadata manager is used to maintain global data index information, which is used for data query;

[0074] The adaptive cache manager is used to predict data access patterns and dynamically change the cache strategy based on the predicted data access patterns, changing the data cached in the node, thereby improving data query efficiency.

[0075] Each core component can perform data scheduling on data in any node in the three storage layers. The specific implementation steps of each core component are described in detail in the subsequent embodiments.

[0076] Specifically, the data flow processing process is as follows: After data generation, the data to be stored undergoes data processing. Specifically, the intelligent data classifier assigns the generated data to the corresponding storage tier. The processed data is then stored. Specifically, the distributed metadata manager updates data index information based on the stored data. The adaptive cache manager predicts access patterns based on each node's historical access history and adjusts the cache policy. Finally, data access is performed. The dynamic data scheduler schedules data across different storage nodes at preset intervals to fully utilize the node's storage resources.

[0077] Based on the above system architecture, the storage of data to be stored is realized based on intelligent data classifier and local load balancing algorithm. Figure 2 As shown, the method is applied to an electronic device, and the method includes:

[0078] S201: Acquire data to be stored.

[0079] The data to be stored may be data generated by edge devices or applications; it is usually a data item newly generated or newly written into the system, and the data item may be a single data object, a file, or a data unit.

[0080] S202: Input the data to be stored into the intelligent classifier to obtain the target storage layer output by the intelligent classifier.

[0081] The target storage layer is used to store the data to be stored. The intelligent classifier is the aforementioned intelligent data classifier. The intelligent classifier is implemented based on a machine learning model.

[0082] S203: Obtain the preset weight of each storage node in the target storage layer, poll each storage node according to the preset weight corresponding to each storage node, and determine the target storage node.

[0083] Among them, the preset weight is determined based on the actual load situation of each storage node. The actual load situation can be determined based on the preset indicator characteristics of each storage node. The preset indicator characteristics are determined based on actual business needs. For example, the preset indicator characteristics can be CPU usage, memory usage, etc.

[0084] Specifically, the polling method for storage nodes can be implemented as follows:

[0085] Obtain the amount of data to be stored, and for each storage node, calculate the ratio of the preset weight of the storage node to the total value of the preset weights of all storage nodes in the target storage layer to obtain a first ratio, and then calculate the product of the number of data to be stored and the first ratio to obtain the target number of data stored by the storage node, randomly select the target number of data from the data to be stored, and store the selected data in the storage node.

[0086] S204: Store the data to be stored in the target storage node, and generate a corresponding relationship between the target storage node and the data to be stored.

[0087] Using the above method, after acquiring the data to be stored, an intelligent classifier is used to determine the target storage tier for the data. This allows for tiered data storage by distinguishing between different data storage tiers, effectively utilizing edge computing resources. Within the target storage tier, polling is performed based on the preset weights of each storage node within the target storage tier to determine the target storage node for the data to be stored. This allows for dynamic load balancing within the target storage tier based on the preset weights, improving overall data storage performance.

[0088] The following combination Figure 3 Introduce the process of training to obtain intelligent classifiers, such as Figure 3 As shown, before S102, inputting the data to be stored into the intelligent classifier and obtaining the target storage layer output by the intelligent classifier, the method includes:

[0089] S301. Input the training data into the time feature extraction network, content feature extraction network and access pattern feature extraction network in the neural network model respectively, and obtain the time series features extracted by the time feature extraction network, the content features extracted by the content feature extraction network and the access pattern features extracted by the access pattern feature extraction network.

[0090] Among them, the BERT model can be used to extract content features, the machine learning model can be used to extract access pattern features, and the Long Short-Term Memory (LSTM) network can be used to extract time series features.

[0091] S302: Combine the time series features, content features, and access pattern features through a neural network model to obtain a target vector.

[0092] In an example, if the time series feature is F time , the content feature is F conten , the access pattern characteristic is F access , then the concatenated target vector can be expressed as:

[0093] S303: Input the target vector into the multi-layer perceptron in the neural network model to obtain a prediction result.

[0094] Among them, the multi-layer perceptron calculates the probabilities corresponding to different storage layers through the activation function. Specifically, the prediction results are calculated according to the following formula:

[0095] y=softmax(W2·RELU(W1·F+b1)+b2

[0096] Among them, W1, W2, b1 and b2 are the model parameters of the multilayer perceptron, and the model parameters can be adjusted later through the loss function.

[0097] S304: Calculate the loss function value of the label data and the prediction result.

[0098] Specifically, the cross entropy loss function can be used to calculate the loss function value. The loss function formula is as follows:

[0099] L=-∑(t i log(y i ))

[0100] Among them, t i Indicates the i-th label data, y i is the i-th prediction result.

[0101] S305. Use the loss function value to adjust the model parameters of the neural network model to obtain an intelligent classifier.

[0102] Using the method provided in the embodiments of the present application, time series features, content features, and access pattern features are extracted through the time feature extraction network, content feature extraction network, and access pattern feature extraction network in the neural network model. The obtained time series features, content features, and access pattern features are then concatenated into a target vector. This target vector can represent the data features from multiple dimensions. Furthermore, a multilayer perceptron is used to obtain a prediction result. Based on the data features from multiple dimensions, the multilayer perceptron can more accurately predict the storage layer corresponding to the data.

[0103] The following combination Figure 4 Introduce the working process of the above intelligent classifier, such as Figure 4 As shown, the method includes:

[0104] S401: Receive data.

[0105] The data is data to be stored.

[0106] S402: Extract features from the data.

[0107] The feature extraction includes S403 to S405. S403 to S405 are executed in parallel to obtain time series features, content features, and access pattern features respectively.

[0108] S403: Extract time series features.

[0109] S404: Extract content features.

[0110] S405: Extract access pattern features.

[0111] S406: Perform feature fusion.

[0112] The feature fusion is a process in which the electronic device in the above embodiment combines the above extracted features to obtain a target vector.

[0113] S407: Make a classification decision based on the fused features.

[0114] Specifically, a multi-layer perceptron is used for classification decision making.

[0115] S408: Return to the storage layer decision.

[0116] Using the method provided in the embodiments of this application, the time feature extraction network, content feature extraction network, and access pattern feature extraction network in the intelligent classifier are used to extract time series features, content features, and access pattern features, respectively. The obtained time series features, content features, and access pattern features are then concatenated into a target vector. This target vector can represent the data features from multiple dimensions. Furthermore, a multilayer perceptron is used to obtain prediction results. Based on the data features from multiple dimensions, the multilayer perceptron can more accurately predict the storage layer corresponding to the data.

[0117] After determining the target storage layer, you can also use the local load balancing mechanism to achieve load balancing of different storage nodes in the storage layer, such as Figure 5 As shown, Figure 5 Three storage nodes and three clients are shown as an example.

[0118] Among them, after the load balancer receives requests sent by different clients, the load balancer obtains the load status of each storage node through load monitoring, and then determines the preset weight corresponding to each storage node according to the load status of each storage node. The load balancer dynamically adjusts the storage nodes corresponding to different requests according to the preset weight. Figure 5 As shown, the load balancer assigns request 1 sent by client 1 to storage node 1; assigns request 2 sent by client 2 to storage node 2; and assigns request 3 sent by client 3 to storage node 3.

[0119] After the data is stored in the corresponding storage node, the distributed data manager is updated to update the data index information. Specifically, the distributed data manager can maintain a distributed hash table. The distributed hash table is used to query the storage nodes corresponding to different data. Based on this, for S104, the data to be stored is stored in the target storage node, and the corresponding relationship between the target storage node and the data to be stored is generated, such as Figure 6 As shown, it can be implemented as follows:

[0120] S601: Obtain data identification of data to be stored.

[0121] The data to be stored includes the data identifier, which may be a file name or an object ID, etc. There is no limitation here, as long as it can uniquely identify the data to be stored.

[0122] S602: Calculate a first hash value corresponding to the data identifier according to a preset hash algorithm.

[0123] S603: Map the first hash value to a preset hash ring to obtain a first hash position.

[0124] S604 : For each virtual storage node corresponding to the target storage node, calculate a second hash value of the virtual storage node according to a preset hash algorithm.

[0125] There are multiple virtual storage nodes corresponding to each storage node. The more virtual storage nodes there are, the more second hash values are calculated, and the more positions are occupied on the preset hash ring, so that the storage node can carry more data.

[0126] It should be noted that, in the embodiments of the present application, the method for calculating the hash value is the same hash algorithm.

[0127] S605: Map the second hash value to a preset hash ring to obtain a second hash position corresponding to each virtual storage node.

[0128] S606: Determine a target hash position that is the same as the first hash position in the second hash position corresponding to each virtual storage node.

[0129] S607: Construct a corresponding relationship between the data identifier, the target hash position, and the target storage node.

[0130] Here, the corresponding relationship is the data index information in the above embodiment.

[0131] By using the method provided in the embodiment of the present application, by calculating the first hash value of the data identifier and the second hash value of the virtual storage node, the first hash position corresponding to the data to be stored and the second hash position corresponding to the virtual storage node can be mapped respectively, and then the correspondence between the rough node and the data identifier can be constructed through the overlapping target hash positions. In this way, the electronic device can maintain the correspondence between the data identifier and the storage node. When accessing data subsequently, the data to be accessed can be quickly found according to the correspondence, thereby improving data access efficiency.

[0132] It should be noted that after the above-mentioned construction obtains the corresponding relationship between the data identifier and the storage node, that is, after the data index information is determined, the distributed metadata manager can achieve fast access to the data based on the data index information. Figure 7As shown, the method includes:

[0133] S701: Determine the operation type.

[0134] If the operation type is a read operation, S702 is executed; if the operation type is a write operation, S707 is executed.

[0135] S702: Check the local cache.

[0136] Part of the data is cached in the cache space of each storage node. The part of the data is determined by the adaptive cache management based on the cache algorithm. The specific method of determining the data in the cache space will be described in detail in subsequent embodiments.

[0137] S703: Determine whether the cache is hit.

[0138] If yes, execute S706; if no, execute S704.

[0139] S704: Query the distributed hash table.

[0140] S705: Update local cache.

[0141] When a read operation occurs, the adaptive cache manager can adjust the current cache algorithm based on the access situation, thereby updating the data cached in the storage node.

[0142] S706. Return the result.

[0143] S707. Use the Raft algorithm to reach consensus.

[0144] Among them, the Raft algorithm is used to maintain data format consistency.

[0145] S708: Update the distributed hash table.

[0146] S709: Clear related cache.

[0147] S710: Return the operation result.

[0148] By adopting the method provided in the embodiment of the present application, the distributed metadata manager can realize fast positioning and fast access of data based on the distributed hash table. After a read operation occurs, since the access situation of different nodes changes, in order to effectively utilize the cache resources in the storage node, the cache data in the cache space can be dynamically adjusted based on the adaptive cache manager. In this way, the efficiency of data reading can be improved.

[0149] In some embodiments of the present application, the storage business load of each physical storage node can be adjusted by adjusting the number of virtual storage nodes corresponding to each physical storage node, thereby achieving global load balancing in the entire system. Specifically, the above data storage method also includes:

[0150] Step 1: Obtain key indicator information for each storage node.

[0151] The key indicator information is pre-set according to actual business needs. For example, the key indicator information may be indicator information such as CPU usage, memory usage, and network throughput.

[0152] It should be noted that the electronic device can obtain key indicator information of the storage node every preset period and perform global load balancing.

[0153] Step 2: For each storage node, determine the actual load information of the storage node based on the key indicator information.

[0154] Step 3: For each storage node, the number of virtual storage nodes corresponding to the storage node is adjusted according to the actual load information and the preset step size. The virtual storage nodes are used to store data.

[0155] Using the method provided in the embodiments of the present application, after protecting the key indicator information of each storage node, the actual load of each storage node can be determined based on the key indicator information. In this way, when the actual load of the storage node is high, the number of virtual storage nodes can be reduced, and when the actual load of the storage node is low, the number of virtual storage nodes can be increased. In this way, by adjusting the number of virtual storage nodes, the load of each storage node is adjusted, achieving load balancing for each node in the system.

[0156] It should be noted that after storing the data in the corresponding storage node, the electronic device can also call the dynamic data scheduler to schedule specific data in the node according to the real-time network status and the actual status of each node, thereby reducing the overall data scheduling delay. Among them, the dynamic data scheduler can use the deep Q-value learning network (DeepQ-Network, DQN) to make scheduling decisions. Figure 8 As shown, the method further includes:

[0157] S801. Obtain status information and network information of each storage node.

[0158] The status information includes the load of each storage node, available storage space, network link latency and bandwidth, and access frequency or access pattern of recent historical data.

[0159] S802: Construct system status representation information according to the status information and network information.

[0160] In an example, the constructed system status representation information can be expressed as:

[0161]

[0162] Among them, load balance Indicates the storage node load, storage usage Indicates available storage space, network condition Indicates network delay and bandwidth information, data popularity Indicates the data access mode.

[0163] S803. For target data in each storage node that meets preset scheduling conditions, calculate the instant rewards of the target data under different actions under the system state representation information according to a preset reward function.

[0164] Among them, the preset scheduling conditions can be set according to actual business needs. The actions include migration or replication, specifically including migration to the edge layer, migration to the regional layer, migration to the central meta layer and replication. Different actions can be represented as action space A = move to_edge ,move to_regional ,move to_cloud ,replicate,do nothing .

[0165] The reward function is pre-built, for example the reward function can be

[0166] Among them, the Q value can be updated according to the learning rate and discount factor. Specifically, Q(s,a)←Q(s,a)+α[R+γmax α′ Q(s′,a′)-Q(s,a).

[0167] Among them, α is the learning rate and γ is the discount factor.

[0168] After the reward function is constructed as above, based on - Greedy strategy explores the action space and determines the best action. Specifically:

[0169]

[0170] in, is the exploration probability, |A| is the action space.

[0171] S804: Schedule target data according to the immediate reward.

[0172] S805: When the target data is scheduled, generate a corresponding relationship between the target data and the storage node.

[0173] By adopting the method provided in the embodiment of the present application, after obtaining the status information and network information of the storage node, the overall status representation of the system is constructed based on the status information. The target data in each storage node that meets the preset scheduling conditions can be determined through the system status representation information. For the target data, the system status and different scheduling actions are comprehensively considered to obtain the instant reward that can be obtained for the target data under different scheduling actions. The instant reward can reflect the improvement of system performance. In this way, the target data can be scheduled according to the timely rewards corresponding to different actions to achieve performance optimization of the entire system.

[0174] The following combination Figure 9 Introduce the workflow of data scheduling by the above dynamic data scheduler, such as Figure 9 As shown, the method includes:

[0175] S901. Monitor system status.

[0176] Among them, electronic equipment can monitor in real time Figure 1 The status information and network information corresponding to each storage node in the system architecture.

[0177] S902: Get the current status.

[0178] The electronic device calls the dynamic data scheduler every preset period to obtain the current status information and network information of each storage node.

[0179] S903: Use DQN to select a scheduling action.

[0180] Specifically, the method for the dynamic data scheduler to determine the target data and the corresponding action is described in the above embodiments and will not be repeated here.

[0181] S904: Execute the scheduling action.

[0182] S905. Observe the new status and reward.

[0183] S906: Update the DQN model.

[0184] Among them, after obtaining new system state information and immediate rewards, the Q value in the reward function is updated, thereby updating the DQN model.

[0185] By adopting the method provided in the embodiment of the present application, after obtaining the status information and network information of the storage node, the overall status representation of the system is constructed based on the status information. The target data in each storage node that meets the preset scheduling conditions can be determined through the system status representation information. By comprehensively considering the system status and different scheduling actions, the instant reward that can be obtained for the target data under different scheduling actions can be calculated. The instant reward can reflect the improvement of system performance. In this way, the target data can be scheduled according to the timely rewards corresponding to different actions, so as to achieve performance optimization of the entire system.

[0186] In some embodiments of the present application, the adaptive cache manager can also adjust the local cache algorithm according to the access mode of the storage node, select the appropriate cache algorithm, and then optimize the storage space corresponding to the different types of data stored in each storage node, thereby optimizing the overall storage space and improving the speed of data reading. Figure 10 As shown, the method further includes:

[0187] S1001. For each storage node, obtain historical access data of the storage node.

[0188] The historical access data may be access logs, access frequencies, or access time series, etc. This application does not impose any restrictions on this.

[0189] S1002: Input historical access data into an access pattern prediction model to obtain a predicted access pattern.

[0190] The predicted access pattern refers to the data access trends corresponding to the storage node within a preset time period, for example, whether users are concentratedly accessing a certain type of data on the storage node. The access pattern prediction model is implemented based on a time series analysis network and machine learning algorithms.

[0191] Specifically, the predicted access pattern can be the data block ID that is most likely to be accessed within a preset time period in the future, or the type of access pattern, for example, the access pattern can be sequential access, random access, or high-frequency access, or the probability of each data block being accessed.

[0192] S1003 : Determine a target cache algorithm corresponding to the predicted access pattern according to a preset correspondence between the access pattern and the cache algorithm.

[0193] In the case where the predicted access pattern is an access pattern type, the corresponding target cache algorithm may be determined based on a preset correspondence between the access pattern type and the cache algorithm.

[0194] When the predicted access pattern is the data block ID that is most likely to be accessed within a preset time period in the future or the probability of each data block being accessed, the data features in the prediction results can be extracted, such as the time limitation, frequency limitation or mixed pattern of the data features, and then the corresponding target cache algorithm can be determined based on the correspondence between the characteristics presented by the data features and the cache algorithm.

[0195] Among them, the cache algorithm can be the Least Recently Used (LRU) algorithm, the Least Frequently Used (LFU) algorithm, or the Adaptive Replacement Cache (ARC) algorithm. The above cache algorithms are only used as examples. In actual implementation, they can be pre-set according to actual business needs.

[0196] S1004: Adjust the cache space of different types of data in the storage node based on the target cache algorithm.

[0197] Since the importance and access frequency of different types of data change dynamically, in order to optimize the limited cache resources on each storage node and improve the utilization efficiency of cache resources, the cache space of different types of data can be dynamically adjusted.

[0198] Specifically, online learning algorithms can be used to obtain real-time data access information for each storage node, allowing for real-time adjustments to the cache allocation for different data types. For example, based on access information, the cache allocation for frequently accessed data can be increased, while the cache allocation for less frequently accessed data can be reduced.

[0199] Using the method provided in the embodiments of the present application, after obtaining the historical access data of each storage node, the predicted access pattern of each storage node in the future can be predicted based on the historical access data and the access pattern prediction model. Different access patterns correspond to different cache algorithms. The corresponding target cache algorithm is determined based on the predicted access pattern. Finally, the cache space for different types of data is adjusted based on the target cache algorithm. In this way, the cache algorithm of each storage node can be dynamically adjusted based on the changing trend of the access pattern of each storage node, and storage resources can be allocated to each storage node.

[0200] The following combination Figure 11 Introduce the specific workflow of the adaptive cache manager, such as Figure 11 As shown, the method includes:

[0201] S1101. Monitor data access patterns.

[0202] The data access pattern refers to user access characteristics, which have time locality or access frequency locality.

[0203] S1102 . Predict data access patterns.

[0204] Data access patterns can be predicted through time series analysis and machine learning algorithms. Specifically, the input to the machine learning model can be historical access data within a preset time period, such as access logs, access frequency, and access time series.

[0205] The output of the machine learning model can be the data block ID that is most likely to be accessed in the future, or the access pattern type, such as sequential access, random access, or high-frequency access, or the access probability corresponding to each data block. This embodiment of the present application does not impose any restrictions on this.

[0206] S1103: Select an optimal cache strategy.

[0207] The electronic device can determine the optimal cache strategy based on the output of the machine learning model. The cache strategy can be LRU, LFU, or ARC.

[0208] S1104: Dynamically adjust cache space allocation.

[0209] Electronic devices can use online learning algorithms to adjust model parameters based on real-time data to predict the cache ratio occupied by different data categories. They can then adjust the cache space allocated to different data categories based on the prediction results. For example, electronic devices can dynamically increase the cache space for frequently accessed data categories and reduce the cache space for less frequently accessed data categories.

[0210] S1105: Apply the new cache strategy.

[0211] It can be understood that after the optimal cache strategy is determined as described above, the cached data is changed according to the optimal cache strategy.

[0212] By adopting the method provided in the embodiment of the present application, after obtaining the historical access data of each storage node, the adaptive cache manager can predict the predicted access pattern of each storage node in the future period based on the historical access data and the access pattern prediction model, wherein different access patterns correspond to different cache algorithms, and the corresponding target cache algorithm is determined according to the predicted access pattern. In this way, the cache algorithm of each storage node can be dynamically adjusted according to the changing trend of the access pattern of each storage node, and the storage resource allocation of each storage node can be realized.

[0213] After the data storage is completed, based on Figure 1The system architecture shown in FIG. 1 shows a system architecture in which users can request to access data stored in a storage node in the system architecture. Specifically, Figure 12 Introduces the complete process of data storage, data access, and data scheduling, such as Figure 12 As shown, the method specifically includes a data entry process, a data access process and a data scheduling process.

[0214] Among them, the data entry process includes S1201 to S1204.

[0215] S1201: Data generation.

[0216] After the edge device or application generates data, the generated data is input into the intelligent data classifier for data classification.

[0217] It should be noted that the data processed by the intelligent data classifier can be a single data object, a file or a data unit.

[0218] S1202. Data classification.

[0219] Among them, the electronic device can classify data through an intelligent data classifier.

[0220] S1203: Data storage.

[0221] S1204: Metadata update.

[0222] The metadata refers to the data index information in the above embodiment.

[0223] The data access process includes S1205 to S1208.

[0224] S1205: Receive request.

[0225] The request carries the data identifier of the data to be accessed.

[0226] S1206: Metadata query.

[0227] Wherein, based on the above data identification and data index information, the corresponding storage node can be found.

[0228] S1207. Obtain data.

[0229] S1208: Update cache.

[0230] After a read operation occurs, the cache algorithm may be updated based on the adaptive cache management. After the cache algorithm is updated, the data in the cache space of the corresponding storage node changes.

[0231] The data scheduling process includes S1209 to S1212.

[0232] S1209: Scheduling decision.

[0233] S1210: Data migration or replication.

[0234] S1211. Metadata update.

[0235] S1212, system monitoring.

[0236] By adopting the method provided in the embodiment of the present application, after data storage, when a data access request sent by a user is received, the storage location of the data to be accessed can be found based on the data identifier carried in the request and the data index information corresponding to the storage node, thereby determining the corresponding storage node. In this way, rapid positioning and rapid reading of the data to be accessed are achieved. When the data access situation changes, the adaptive cache manager can update the cached data in a timely manner, reducing the delay of subsequent data access. In addition, the dynamic data scheduler can make scheduling action decisions for the data in each storage node in real time, thereby optimizing the storage location of the data, reducing the overall delay of the above-mentioned system architecture, and improving system performance.

[0237] In some embodiments of the present application, since node failure is unavoidable in the distributed system shown in the figure, Figure 1 The system architecture shown also includes a fault detector for fault detection and fault recovery.

[0238] like Figure 13 As shown, Figure 13 The example shows two healthy nodes and one faulty node. The fault detector regularly performs heartbeat checks on each storage node based on a heartbeat mechanism to identify the faulty node. Upon detecting a faulty node, the fault detector sends a command to the recovery manager, instructing it to initiate recovery for the faulty node. After receiving the command from the fault detector, the recovery manager initiates a circuit breaker on the faulty node, rapidly isolating it. The recovery manager then migrates the data stored in the faulty node to a healthy node.

[0239] During the data migration process, the Raft consensus algorithm and Write-Ahead Logging (WAL) are used to ensure the data consistency of the migrated data and local data.

[0240] Specifically, the automatic recovery process is as follows: first isolate the faulty node, then reallocate tasks for the assigned requests, migrate the data in the faulty node to the normal node, and when the faulty node is rectified, migrate the migrated data back to the node, restore data consistency, and rejoin the node to the normal node cluster of the storage layer.

[0241] Based on the same concept, the embodiment of the present application provides a data storage device, which is applied to electronic devices, such as Figure 14 As shown, the device includes:

[0242] An acquisition module 1401 is used to acquire data to be stored;

[0243] A classification module 1402 is configured to input the data to be stored into an intelligent classifier, and obtain a target storage layer output by the intelligent classifier, wherein the target storage layer is used to store the data to be stored;

[0244] The polling module 1403 is configured to obtain a preset weight of each storage node in the target storage layer, and poll each storage node according to the preset weight corresponding to each storage node to determine the target storage node;

[0245] The storage module 1404 is configured to store the data to be stored in the target storage node and generate a corresponding relationship between the target storage node and the data to be stored.

[0246] In a possible implementation, the device further includes:

[0247] a feature extraction module, configured to input the training data into the time feature extraction network, the content feature extraction network, and the access pattern feature extraction network in the neural network model, respectively, to obtain time series features extracted by the time feature extraction network, content features extracted by the content feature extraction network, and access pattern features extracted by the access pattern feature extraction network;

[0248] a splicing module, configured to splice the time series features, the content features, and the access pattern features through the neural network model to obtain a target vector;

[0249] The classification module 1402 is further configured to input the target vector into the multilayer perceptron in the neural network model to obtain the prediction result;

[0250] A calculation module, used to calculate the loss function value of the label data and the prediction result;

[0251] An adjustment module is used to adjust the model parameters of the neural network model using the loss function value to obtain the intelligent classifier.

[0252] In a possible implementation, the storage module 1404 is specifically configured to:

[0253] Obtaining a data identifier of the data to be stored;

[0254] Calculate a first hash value corresponding to the data identifier according to a preset hash algorithm;

[0255] Mapping the first hash value to a preset hash ring to obtain a first hash position;

[0256] For each virtual storage node corresponding to the target storage node, calculating a second hash value of the virtual storage node according to the preset hash algorithm;

[0257] Mapping the second hash value to the preset hash ring to obtain a second hash position corresponding to each virtual storage node;

[0258] Determine, in the second hash position corresponding to each virtual storage node, a target hash position that is the same as the first hash position;

[0259] A correspondence between the data identifier, the target hash position, and the target storage node is constructed.

[0260] In a possible implementation, the device further includes:

[0261] The acquisition module 1401 is further used to obtain key indicator information of each storage node;

[0262] A determination module, configured to determine, for each storage node, actual load information of the storage node according to the key indicator information;

[0263] The adjustment module is configured to adjust, for each storage node, the number of virtual storage nodes corresponding to the storage node according to the actual load information and a preset step size, wherein the virtual storage nodes are used to store data.

[0264] In a possible implementation, the device further includes:

[0265] The acquisition module 1401 is further used to obtain status information and network information of each storage node;

[0266] A construction module, configured to construct system status representation information according to the status information and the network information;

[0267] The calculation module is further configured to calculate, for each storage node, target data that meets preset scheduling conditions, an instant reward for the target data under different actions under the system state representation information according to a preset reward function;

[0268] a scheduling module, configured to schedule the target data according to the instant reward;

[0269] The storage module 1404 is further configured to generate a correspondence between the target data and the storage node when the target data is scheduled.

[0270] In a possible implementation, the method further includes:

[0271] The acquisition module 1401 is further configured to acquire, for each storage node, historical access data of the storage node;

[0272] A prediction module, configured to input the historical access data into an access pattern prediction model to obtain a predicted access pattern;

[0273] A determination module, configured to determine a target cache algorithm corresponding to the predicted access pattern based on a preset correspondence between the access pattern and the cache algorithm;

[0274] An adjustment module is used to adjust the cache space of different types of data in the storage node based on the target cache algorithm.

[0275] It should be noted that the data storage device is a device corresponding to the above-mentioned data storage method. All implementation methods in the above-mentioned method embodiments are applicable to the embodiments of the device and can achieve the same technical effects.

[0276] Figure 15 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application is shown.

[0277] The electronic device may include a processor 1501 and a memory 1502 storing computer program instructions.

[0278] Specifically, the processor 1501 may include a central processing unit (CPU) or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present application.

[0279] Memory 1502 may include a large capacity memory for data or instructions. By way of example and not limitation, memory 1502 may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a universal serial bus (USB) drive, or a combination of two or more of these. Where appropriate, memory 1502 may include removable or non-removable (or fixed) media. Where appropriate, memory 1502 may be inside or outside the integrated gateway disaster recovery device. In a specific embodiment, memory 1502 is a non-volatile solid-state memory.

[0280] In certain embodiments, the memory 1502 may include a read-only memory (ROM), a random access memory (RAM), a magnetic disk storage medium device, an optical storage medium device, a flash memory device, an electrical, optical, or other physical / tangible memory storage device. Thus, generally, the memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., a memory device) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform the operations described with reference to the method according to an aspect of the present disclosure.

[0281] The processor 1501 implements any one of the data storage methods in the above embodiments by reading and executing computer program instructions stored in the memory 1502 .

[0282] In one example, the electronic device may further include a communication interface 1503 and a bus 1504. Figure 15 As shown, the processor 1501, the memory 1502, and the communication interface 1503 are connected via a bus 1504 and communicate with each other.

[0283] The communication interface 1503 is mainly used to implement communication between various modules, devices, units and / or equipment in the embodiments of the present application.

[0284] Bus 1504 includes hardware, software, or both that couples components of an electronic device to each other. By way of example and not limitation, a bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Extended Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a Super Transmission (HT) interconnect, an Industrial Standard Architecture (ISA) bus, an InfiniBand interconnect, a Low Pin Count (LPC) bus, a memory bus, a Micro Channel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local Bus (VLB) bus, or other suitable buses, or a combination of two or more of these. Where appropriate, bus 1504 may include one or more buses. Although embodiments herein describe and illustrate a particular bus, this application contemplates any suitable bus or interconnect.

[0285] In addition, in conjunction with the data storage methods in the above embodiments, embodiments of the present application may provide a computer storage medium for implementation. The computer storage medium stores computer program instructions; when the computer program instructions are executed by a processor, any of the data storage methods in the above embodiments is implemented.

[0286] An embodiment of the present application further provides a computer program product, including a computer program, which implements any one of the data storage methods in the above embodiments when the computer program is processed and executed.

[0287] It should be understood that the present application is not limited to the specific configurations and processes described above and illustrated in the figures. For the sake of brevity, a detailed description of known methods is omitted here. In the above embodiments, several specific steps are described and illustrated as examples. However, the method process of the present application is not limited to the specific steps described and illustrated. Those skilled in the art can make various changes, modifications, and additions, or change the order of the steps after understanding the spirit of the present application.

[0288] The functional blocks shown in the above block diagram can be implemented as hardware, software, firmware or a combination thereof. When implemented in hardware, it can be, for example, an electronic circuit, an application specific integrated circuit (ASIC), appropriate firmware, a plug-in, a function card, etc. When implemented in software, the elements of the present application are programs or code segments that are used to perform the required tasks. The program or code segment can be stored in a machine-readable medium, or transmitted on a transmission medium or a communication link via a data signal carried in a carrier wave. "Machine-readable medium" can include any medium that can store or transmit information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (erasable read-only memory, EROM), floppy disks, compact disc read-only memory (Compact Disc Read-Only Memory, CD-ROM), optical discs, hard disks, optical fiber media, radio frequency (Radio Frequency, RF) links, etc. The code segment can be downloaded via a computer network such as the Internet, an intranet, etc.

[0289] It should also be noted that the exemplary embodiments mentioned in this application describe some methods or systems based on a series of steps or devices. However, this application is not limited to the order of the above steps. In other words, the steps can be performed in the order mentioned in the embodiments, or in a different order, or several steps can be performed simultaneously.

[0290] Aspects of the present disclosure have been described above with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer or other programmable data processing device to produce a machine so that these instructions executed by the processor of the computer or other programmable data processing device enable the implementation of the function / action specified in one or more boxes of the flowchart and / or block diagram. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor or a field programmable logic circuit. It is also understood that each box in the block diagram and / or flowchart and the combination of the boxes in the block diagram and / or flowchart can also be implemented by dedicated hardware that performs the specified function or action, or can be implemented by a combination of dedicated hardware and computer instructions.

[0291] The above is only a specific implementation method of the present application. Those skilled in the art can clearly understand that for the convenience and brevity of description, the specific working processes of the systems, modules and units described above can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here. It should be understood that the scope of protection of the present application is not limited to this. Any technician familiar with this technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in this application, and these modifications or replacements should be included in the scope of protection of this application.

Claims

1. A data storage method, characterized in that: Applied to electronic equipment, the method includes: Get the data to be stored; Inputting the data to be stored into an intelligent classifier to obtain a target storage layer output by the intelligent classifier, wherein the target storage layer is used to store the data to be stored; Obtaining a preset weight of each storage node in the target storage layer, and polling each storage node according to the preset weight corresponding to each storage node to determine the target storage node; The data to be stored is stored in the target storage node, and a corresponding relationship between the target storage node and the data to be stored is generated.

2. The method according to claim 1, characterized in that Before inputting the data to be stored into an intelligent classifier to obtain a target storage layer output by the intelligent classifier, and the target storage layer is used to store the data to be stored, the method further includes: Inputting the training data into the time feature extraction network, content feature extraction network and access pattern feature extraction network in the neural network model respectively, obtaining the time series features extracted by the time feature extraction network, the content features extracted by the content feature extraction network and the access pattern features extracted by the access pattern feature extraction network; The time series features, the content features, and the access pattern features are combined by the neural network model to obtain a target vector; Inputting the target vector into the multilayer perceptron in the neural network model to obtain a prediction result; Calculating a loss function value for the label data and the prediction result; The loss function value is used to adjust the model parameters of the neural network model to obtain the intelligent classifier.

3. The method according to claim 1, characterized in that The storing the data to be stored in the target storage node and generating a corresponding relationship between the target storage node and the data to be stored includes: Obtaining a data identifier of the data to be stored; Calculate a first hash value corresponding to the data identifier according to a preset hash algorithm; Mapping the first hash value to a preset hash ring to obtain a first hash position; For each virtual storage node corresponding to the target storage node, calculating a second hash value of the virtual storage node according to the preset hash algorithm; Mapping the second hash value to the preset hash ring to obtain a second hash position corresponding to each virtual storage node; Determine, in the second hash position corresponding to each virtual storage node, a target hash position that is the same as the first hash position; A correspondence between the data identifier, the target hash position, and the target storage node is constructed.

4. The method according to claim 1, wherein Also includes: Obtain key indicator information for each storage node; For each storage node, determining actual load information of the storage node according to the key indicator information; For each storage node, the number of virtual storage nodes corresponding to the storage node is adjusted according to the actual load information and a preset step size, and the virtual storage nodes are used to store data.

5. The method according to claim 1, wherein Also includes: Obtain status information and network information of each storage node; Constructing system status representation information according to the status information and the network information; For target data in each storage node that meets preset scheduling conditions, calculate the instantaneous reward of the target data under different actions under the system state representation information according to a preset reward function; Scheduling the target data according to the instant reward; When the target data is scheduled, a corresponding relationship between the target data and the storage node is generated.

6. The method according to claim 1, characterized in that Also includes: For each storage node, obtaining historical access data of the storage node; Inputting the historical access data into an access pattern prediction model to obtain a predicted access pattern; Determining a target cache algorithm corresponding to the predicted access pattern based on a preset correspondence between the access pattern and the cache algorithm; The cache space of different types of data in the storage node is adjusted based on the target cache algorithm.

7. A data storage device, characterized in that: Applied to electronic equipment, the device comprises: An acquisition module, used to acquire data to be stored; a classification module, configured to input the data to be stored into an intelligent classifier, and obtain a target storage layer output by the intelligent classifier, wherein the target storage layer is used to store the data to be stored; A polling module is used to obtain a preset weight of each storage node in the target storage layer, and poll each storage node according to the preset weight corresponding to each storage node to determine the target storage node; The storage module is used to store the data to be stored in the target storage node and generate a corresponding relationship between the target storage node and the data to be stored.

8. An electronic device, characterized in that: The device includes: a processor and a memory storing computer program instructions; When the processor executes the computer program instructions, the data storage method according to any one of claims 1 to 6 is implemented.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer program instructions, which, when executed by a processor, implement the data storage method according to any one of claims 1 to 6.

10. A computer program product, characterized in that When the instructions in the computer program product are executed by a processor of an electronic device, the electronic device executes the data storage method according to any one of claims 1 to 6.

Images