A storage method for a computing power service data set

Abstract

The present invention discloses a method for storing a computing power service data set, which relates to electrical digital data processing and includes classifying the data to be stored and extracting specific features of each data item; determining the data writing order and performing data writing operations in accordance with the order; calculating a hash value for each data item separately after the data writing is completed, and transmitting each data item and its corresponding hash value to a predetermined number of storage nodes for data consistency verification; monitoring the access information of each storage node in real time, dividing logical partitions, and performing scheduling on data access requests according to the load conditions of each storage node; dividing data categories according to data access frequency and data timeliness, and performing data caching operations on the data. This method breaks through the limitations of traditional static scheduling and hierarchical management methods, realizes the intelligence, refinement, and adaptability of data storage and access management, and improves the resource utilization rate and response speed of the computing power service data set storage system.

Description

Technical Field

[0001] The present invention relates to the technical field of electronic digital data processing, and particularly to a method for storing a computing power service data set. Background Art

[0002] In the current fields of computing power services and big data applications, the demand for the generation and storage of massive data has risen rapidly. However, traditional static logical partitioning, fixed scheduling, and simple caching strategies cannot fully adapt to the challenges brought by the dynamic changes in data access frequencies and the load fluctuations of each storage node. At the same time, there is a lack of an effective collaborative management mechanism among independent technologies, resulting in problems such as local optimality or even information islands in the scheduling, classification, and migration operations of data between storage media. The present invention realizes real-time quantitative analysis of data access frequencies, timeliness, and node loads by introducing dynamic mathematical models and machine learning algorithms, breaking the technical bottleneck of the mutual independence of each link in traditional methods as a whole. Summary of the Invention

[0003] In view of the above existing problems, the present invention is proposed.

[0004] Therefore, the present invention proposes an overall data management method that integrates a normalization partitioning function, a weighted node load model, a data classification function, a prediction scheduling model, and a migration function. This method uses mathematical formulas to perform normalization partitioning on data access frequencies, evaluates the load of each storage node in real time through weighted calculation, constructs hot, warm, and cold data classification models based on data access records and timeliness, introduces a prediction function based on time series and machine learning to guide cache scheduling, and uses the migration function to automatically determine the migration decision of data between different storage media, realizing innovative integration and information sharing among each technical link, thus breaking through the static and local optimization problems existing in traditional data scheduling strategies.

[0005] To solve the above technical problems, the present invention provides the following technical solution. A method for storing a computing power service data set includes:

[0006] Classify the data to be stored according to data types and update frequencies, and extract specific features of each data item; determine the data writing order based on the extracted features, historical writing records, and the current system state, and perform data writing operations in accordance with the order; after the data writing is completed, calculate the hash value for each data item separately, and transmit each data item and its corresponding hash value to a predetermined number of storage nodes for data consistency verification; monitor the access information of each storage node in real time, divide all data into a fixed number of logical partitions according to the collected data of access frequencies, response delays, and resource utilization rates, and perform scheduling on data access requests according to the load conditions of each storage node; divide data categories according to data access frequencies and data timeliness, and perform data caching operations on the data.

[0007] As a preferred solution of the computing power service data set storage method described in the present invention, wherein: the extraction of specific features of each data item includes dividing the data into two clear categories of high-speed updated data and low-speed updated data according to the type and update frequency of the data, extracting the specific attributes of each data item for the high-speed updated data and the low-speed updated data respectively, and recording the attribute values of each data item.

[0008] As a preferred solution of the computing power service data set storage method described in the present invention, wherein: the determination of the data writing order includes,

[0009] extracting the specific attributes that make up the feature vector for each data item where represents the th attribute of the th data item and is the total number of attributes, calculating the weighted sum of attributes using a pre-set weight vector , is the weight corresponding to the th attribute, is the weighted sum of the attributes of the data item , through the formula:

[0010] ;

[0011] obtaining the basic score of the data item, introducing a historical writing record function for quantifying the average writing delay or throughput recorded for the th data item in previous operations, and defining the current system state quantity to reflect the overall load cache occupancy rate and response latency of the system at the moment of data writing, and constructing a comprehensive scoring model, where represents the historical writing record function of the th data item, and its value represents the historical writing delay or throughput, is the comprehensive score of the th data item, through the formula:

[0012] ;

[0013] calculating the comprehensive score of the th data item, where and are pre-set coefficients, respectively measuring the influence of data attribute scores, historical records and the current system state on the writing order, and the comprehensive score is used as a prediction of the writing time, that is, let:

[0014] ;

[0015] After the data writing operation is completed, record the actual writing time of each data item And construct a loss function to measure the difference between the predicted value and the actual value and the violation of the order between data items. Define the loss function as:

[0016] ;

[0017] Wherein, represents the predicted writing time according to the comprehensive score, is a balance parameter between 0 and 1, used to trade off between prediction error and order consistency. The set is defined as all pairs of data items that need to satisfy the predetermined writing order constraint, where the order constraint requires that for each pair of must satisfy the condition of ;

[0018] Use the gradient descent method to update the weight parameters in the model online. Its update formula is:

[0019] ;

[0020] Wherein, is the updated weight, is the current weight, is the overall loss function, represents the learning rate, and the update is performed in real time after each data writing operation.

[0021] As a preferred solution of the computing power service data set storage method described in the present invention, wherein: the separate calculation of the hash value includes calculating the hash value of each data item using a fixed algorithm, attaching the calculated hash value to the corresponding data item, and transmitting each data item with the attached hash value to a preset number of storage nodes for subsequent consistency verification.

[0022] As a preferred solution of the computing power service data set storage method described in the present invention, wherein: the data consistency verification includes comparing each received data item and its attached hash value within each storage node, using a distributed consensus algorithm to judge the consistency of the hash values of each data item in all storage nodes, and performing conflict detection on the data item when it is found that the hash value of a certain data item does not match, and performing data synchronization operations on the data item according to the primary and standby synchronization rules.

[0023] As a preferred solution of the computing power service data set storage method described in the present invention, wherein: the real-time monitoring of the access information of each storage node includes that the step of real-time monitoring of the data access information of each storage node includes collecting the data access frequency, response delay and resource utilization rate on each storage node, transmitting the collected access information to the central scheduling module, and recording the transmitted information in the central scheduling module according to a preset data collection period;

[0024] Among them, the logical partitioning of all data according to the monitoring data recorded in the central scheduling module includes counting the data access frequency, dividing the data into a fixed number of logical partitions according to the statistical results, determining the scheduling order of data access requests according to the current load conditions of each storage node, and allocating the data access requests to the corresponding storage nodes according to the scheduling order.

[0025] As a preferred solution of the computing power service data set storage method described in the present invention, wherein: the execution of scheduling for data access requests includes counting the access records of all data items, dividing all data into hot data, warm data and cold data respectively according to the statistical results, storing the divided hot data in a high-speed storage medium, storing the divided cold data in a low-speed storage medium, and performing a predetermined intermediate storage operation on the divided warm data.

[0026] As a preferred solution of the computing power service data set storage method described in the present invention, wherein: the execution of data caching operations includes that the step of periodically evaluating the data access status in each storage medium includes counting the data access frequency in each storage medium within a fixed evaluation period, determining the specific migration scheme of data between different storage media according to the statistical results, and performing the migration operation of data from a low-speed storage medium to a high-speed storage medium, so that the data migration operation and the data caching operation are continuous in the operation process.

[0027] A computer device includes a memory and a processor, the memory stores a computer program, and when the processor executes the computer program, the steps of the computing power service data set storage method are implemented.

[0028] A computer-readable storage medium stores a computer program thereon, and when the computer program is executed by a processor, the steps of the computing power service data set storage method are implemented.

[0029] Advantages of the present invention: By constructing a complete set of dynamic scheduling systems based on mathematical models and machine learning algorithms, the present invention realizes the real-time dynamic allocation of data access requests and storage resources among storage nodes. Its technical solution uses a normalized partitioning function to statistically divide data access frequencies, and combines a weighting function to evaluate indicators such as CPU usage rate, memory occupancy rate, and disk I / O of each node in real time to determine the optimal scheduling order. At the same time, a hot, warm, and cold data classification model is constructed using access records and timeliness data, enabling data to be automatically allocated to high-speed, low-speed, or medium-speed storage media according to access frequencies, and realizing accurate prediction of data caching timing and caching capacity through prediction functions based on time series analysis and machine learning, thereby automatically triggering data migration operations between different storage media. This technology innovatively breaks through the limitations of traditional static scheduling and hierarchical management methods, realizing the intelligent, refined, and adaptive management of data storage and access, and improving the resource utilization rate and response speed of the computing power service data set storage system as a whole. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] To more clearly illustrate the technical solutions of the embodiments of the present invention, the following will briefly introduce the accompanying drawings required for the description of the embodiments. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other accompanying drawings can be obtained based on these drawings.

[0031] Figure 1 It is a schematic flowchart of a method for storing a computing power service data set provided by an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0032] To make the above objects, features, and advantages of the present invention more obvious and understandable, the following will describe the specific embodiments of the present invention in detail with reference to the accompanying drawings of the specification. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0033] Many specific details are set forth in the following description to facilitate a thorough understanding of the present invention, but the present invention may be practiced in other ways different from those described herein. Those skilled in the art can make similar extensions without departing from the connotation of the present invention. Therefore, the present invention is not limited by the specific embodiments disclosed below.

[0034] Embodiment 1

[0035] Referring to Figure 1 , this embodiment provides a method for storing a computing power service data set, including:

[0036] S1: Classify the data to be stored according to the data type and update frequency, and extract the specific features of each data item.

[0037] The extraction of the specific features of each data item includes dividing the data into two categories: high-speed updated data and low-speed updated data according to the data type and update frequency, extracting the specific attributes of each data item for the high-speed updated data and the low-speed updated data respectively, and recording the attribute values of each data item.

[0038] S2: Determine the data writing order based on the extracted features, historical writing records, and the current system state, and perform the data writing operation in accordance with the order.

[0039] Set a group of parameters, which include data format, data source, data generation time interval, and a predefined time sliding window. The length of the time window is determined according to the actual performance of the data processing system and a fixed duration is used as the standard. This duration is determined through previous experiments and is used to preliminarily screen the data when the data enters the processing flow, so as to count the number of updates of each data item within a fixed time. This statistical process uses the moving average calculation method and the standard deviation calculation method to quantitatively analyze the data update frequency. When counting, the moving average window is set to the length of the predefined time sliding window and the standard deviation is calculated using the sample data. The calculation result obtains the dynamic threshold through a formula. The dynamic threshold is determined in the way of adding a fixed multiple of the standard deviation to the moving average value, where the fixed multiple takes the value of 3. This value is determined according to the historical data statistics results to achieve the quantitative division of the data update frequency. The data is divided into high-speed updated data with the number of updates higher than the threshold and low-speed updated data with the number of updates lower than the threshold according to the calculated dynamic threshold. After each type of data is divided, a rule-based parsing algorithm is used for processing. This algorithm uses a preset regular expression template and fixed parsing rules to extract the specific attributes from each data item. These specific attributes are the unique identifier, generation timestamp, data size, data type, and data source information of the data item. The extracted attribute values are stored in a relational database in the form of key-value pairs according to the preset data record structure. An automatic verification module is introduced during storage. This module detects the extraction results according to the predefined data format and numerical range rules, marks the data items that do not meet the format requirements, and removes them according to the filtering conditions.

[0040] Meanwhile, detailed processing logs are generated for the processing status of each data item. The logs record the number of data item updates, the calculated dynamic thresholds, the data classification results, and the specific attribute values extracted. All log information is saved in the central database in a standardized format for subsequent parameter review and system debugging. In addition, a plugin interface is designed and implemented. This interface clearly defines the interface call sequence and data exchange format, allowing users to perform additional attribute parsing operations on data items according to predefined rules in specific application scenarios, and integrating the additionally extracted attributes into the data record in the same format as the main record. All operation steps are executed in sequence according to the fixed process of data preprocessing, update frequency statistics, dynamic threshold calculation, data classification, attribute parsing, automatic verification, data recording, and plugin extension, and detailed records are made of each key parameter and processing information. The entire data processing process is implemented in a continuous operation manner within the data system, and a regular data backup mechanism is used to ensure the complete preservation and real-time update of all data and log information.

[0041] The determination of the data writing order includes

[0042] extracting the specific attributes that make up the feature vector for each data item where represents the th attribute of the th data item and is the total number of attributes. Using the pre-set weight vector calculate the weighted sum of attributes is the weight corresponding to the th attribute, is the weighted sum of the attributes of data item . Through the formula:

[0043] ;

[0044] obtain the basic score of the data item. Introduce the historical write record function to quantify the average write latency or throughput recorded for the th data item in previous operations, and define the current system status quantity to reflect the overall load cache occupancy rate and response latency of the system at the data write moment, and construct a comprehensive scoring model, where represents the historical write record function of the th data item, and its value represents the historical write latency or throughput, is the comprehensive score of the th data item. Through the formula:

[0045] ;

[0046] Calculate the The comprehensive score of a data item, where and are preset coefficients that measure the influence of data attribute scores, historical records, and the current system state on the write order respectively. The comprehensive score is used to predict the write time, that is, let:

[0047] ;

[0048] After the data write operation is completed, record the actual write time of each data item and construct a loss function to measure the difference between the predicted value and the actual value and the violation of the order between data items. Define the loss function as:

[0049] ;

[0050] Among them, represents the write time predicted according to the comprehensive score, is a balance parameter between 0 and 1 used to trade off between prediction error and order consistency. The set is defined as all pairs of data items that need to satisfy the predetermined write order constraint, where the order constraint requires that for each pair must satisfy condition;

[0051] Use the gradient descent method to update the weight parameters in the model online, and its update formula is:

[0052] ;

[0053] Among them, is the overall loss function, represents the learning rate, is the updated weight, The current weight is updated in real time after each data write operation, so that the model can continuously adjust and optimize the prediction results according to the latest actual written data in subsequent operations, thus forming a closed-loop feedback machine learning mechanism. This mechanism integrates multiple links such as data attribute extraction, historical performance records, real-time system state acquisition, comprehensive score calculation, write order prediction, actual write time recording, loss function calculation, and parameter update. Its innovation lies in constructing a comprehensive score by linearly combining the static features of data items, historical behaviors, and the current system environment, and further introducing order constraints into the loss function, thereby ensuring the consistency of the write order while guaranteeing the prediction accuracy. By combining static attributes, historical data, and dynamic system states, this mathematical model determines the data write order using the comprehensive score and adjusts the model parameters through real-time feedback, thus realizing a creative online learning data write sorting mechanism based on the improvement of the existing technology.

[0054] S3: After the data writing is completed, calculate the hash value for each data item separately, and transmit each data item and its corresponding hash value to a predetermined number of storage nodes for data consistency verification.

[0055] In the step of generating the hash value of each data item after the data writing is completed, an improved hash generation mechanism is introduced. This mechanism comprehensively utilizes the characteristics of the data item itself, the dynamic system state, and the verifiable delay function to break through the limitations of traditional fixed hash methods in precomputation and parallel attack prevention. The specific implementation steps are as follows: For each data item Call the secure pseudo-random generation function Get a unique random number ; At the same time, use the deterministic extraction function Extract the fixed signature value from the data item ; After obtaining the current normalized timestamp ; After that, through the following combination function, And Are weighted and fused:

[0056] ;

[0057] Among them, Is the weight coefficient for balancing the random number, Represents the exclusive OR operation;

[0058] Introduce the neural network mapping function Process the current system state to obtain a fixed-length bit string output, and then connect the above combination result with Through the concatenation operator To form an input string:

[0059] ;

[0060] Use a verifiable delay function to calculate the input string , and its output is:

[0061] ;

[0062] This function is designed to have a fixed calculation delay under the condition of no parallel acceleration, thus introducing time constraints in the generated output;

[0063] Combine the original data item with the delayed output through the concatenation operator to form a new data string, and use a fixed cryptographic hash function to calculate the hash value:

[0064] ;

[0065] The generated hash value is the improved hash value, and then attach it to to form an additional data item:

[0066] ;

[0067] Send the additional data item to a predetermined number of storage nodes for subsequent data consistency verification.

[0068] The data consistency verification includes comparing each received data item and its attached hash value within each storage node, using a distributed consensus algorithm to judge the consistency of the hash values of each data item in all storage nodes, and performing conflict detection on the data item when the hash value of a certain data item does not match, and performing data synchronization operations on the data item according to the primary-backup synchronization rule.

[0069] Specifically, during the process of data consistency verification among storage nodes, each storage node receives the data item and its attached hash value , uses a fixed algorithm to verify whether the locally stored matches the attached value, and then each node aggregates the calculated hash values to form (where represents the set of all storage nodes), and uses a distributed consensus mechanism to calculate the consensus hash value of the data item (e.g., by majority voting or a mode algorithm), and its calculation formula is:

[0070] ;

[0071] Wherein, represents the value with the highest frequency of occurrence in the set, and then the average comparison distance is introduced to quantify the difference in hash values between nodes, and its calculation formula is:

[0072] ;

[0073] Wherein, represents the Hamming distance between hash values, is the total number of storage nodes; when exceeds the preset threshold a conflict detection process is triggered, and this process selects the master node according to the primary-backup synchronization rule (which can be determined according to the historical reliability of the nodes or fixed allocation), and calculates the distance between each node and the master node :

[0074] ;

[0075] If for a certain node there is (where is the preset synchronization threshold), then a data synchronization operation is performed to update the data of node to the data of the master node, that is:

[0076] ;

[0077] Wherein, represents the hash value distance between node l and the master node , respectively represent the content of the data item in node l and the master node , represents the hash value calculated or received by storage node l for the data item , represents the hash value corresponding to the data item in the master node P.

[0078] The whole process integrates a consensus mechanism based on taking the mode of hash values, an inconsistency metric of the average comparison distance, and the primary-backup synchronization rule, and realizes a closed-loop process for data item consistency verification among storage nodes through online conflict detection and data synchronization operations.

[0079] S4: Monitor the access information of each storage node in real time, divide all the data into a fixed number of logical partitions according to the collected access frequency, response latency, and resource utilization data, and execute scheduling for data access requests according to the load conditions of each storage node.

[0080] Real-time monitoring of the data access information of each storage node includes collecting the data access frequency, response latency, and resource utilization rate on each storage node, transmitting the collected access information to the central scheduling module, and recording the transmitted information in the central scheduling module according to a preset data collection cycle; the step of logically partitioning all the data according to the monitoring data recorded in the central scheduling module includes counting the data access frequency, dividing all the data into a fixed number of logical partitions according to the statistical results, and clarifying the scheduling order of data access requests according to the current load conditions of each storage node, and allocating the data access requests to the corresponding storage nodes in this order.

[0081] Specifically, after receiving the monitoring data of each storage node, the central scheduling module first counts the data access frequency of each node according to the preset collection cycle, divides all the data into K logical partitions according to the statistical results according to the predefined number of logical partitions K (for example, K = 10), and at the same time collects load indicators such as CPU usage rate, memory occupancy rate, and disk I / O in each storage node and transmits these load information to the central scheduling module, and records and summarizes the data packets transmitted by all nodes in the central scheduling module according to the preset data collection cycle, then sorts the data access requests of each storage node according to the recorded load information and determines the scheduling order, and then allocates the data access requests to the corresponding storage nodes in this order according to the predefined data format; the central scheduling module also counts the historical access records of all data items, divides all the data into hot data, warm data, and cold data according to the statistical results, where the hot data is stored on the high-speed storage medium according to the predefined rules, the cold data is stored on the low-speed storage medium, and the warm data performs the predefined intermediate storage operations, and determines the cache scheduling timing and cache capacity of the data items with higher access frequency according to the historical access records, loads the data to be cached from the storage medium into the cache and performs consistency verification on the loaded data and the original stored data during the loading process; the central scheduling module also counts the data access frequency in each storage medium within a fixed evaluation cycle, determines the specific migration plan of the data between different storage mediums according to the statistical results, and executes the migration operation of the data from the low-speed storage medium to the high-speed storage medium, so that the data migration operation and the data caching operation are continuous in the whole operation process.

[0082] S5: Divide the data categories according to the data access frequency and data timeliness, and perform data caching operations on the data.

[0083] Furthermore, after the monitoring data is collected, the central scheduling module calculates the access frequency of each data item using the formula:

[0084] ;

[0085] to divide the data into a fixed number of logical partitions, where represents the number of accesses of a single data item within the statistical period, represents the maximum access frequency within the statistical period, is the total number of predefined logical partitions, represents the ceiling function;

[0086] The load metrics (including CPU usage, memory occupancy, and disk I / O utilization) collected by each storage node through the monitoring agent are denoted as and respectively. Through the weighted function:

[0087] ;

[0088] the load score of each node l is calculated, where and are preset weight coefficients; the central scheduling module sorts the nodes according to the load scores and allocates the data access requests to the corresponding storage nodes according to the sorting results in the predefined data format, thus determining the scheduling order.

[0089] Meanwhile, the central scheduling module statistically analyzes the historical access records of all data items, records the access frequency of each data item and the data timeliness parameter (such as the time interval since the last access), and according to the classification function:

[0090] ;

[0091] divides the data items into hot data, warm data, and cold data, where and are the hot data and cold data thresholds of the access frequency respectively, and are the timeliness thresholds respectively; according to the data classification results, the hot data is stored in the high-speed storage medium, the cold data is stored in the low-speed storage medium, and the warm data is managed by intermediate storage operations, further refining the data management strategy.

[0092] For data items with a relatively high access frequency, the system introduces a prediction function based on the historical access pattern (which can be implemented through time series models or machine learning algorithms), this function is used to predict future access trends, so as to determine the scheduling timing of data caching and the required cache capacity; once it is predicted that the data access frequency is about to increase, the system will load the data to be cached from the storage medium to the cache according to the predefined protocol, and perform consistency verification on the data during the loading process to ensure that the data content has not deviated during the loading process.

[0093] In addition, within a fixed evaluation period, the central scheduling module counts the data access frequencies in each storage medium, and uses the migration function based on the statistical data:

[0094] ;

[0095] to determine whether it is necessary to perform a data migration operation, where is a preset migration threshold; if the data access frequency reaches or exceeds , the system will automatically trigger a data migration operation to migrate the data from the low-speed storage medium to the high-speed storage medium, and maintain the consistency of the data during the migration process to ensure the continuity of caching and migration of the data throughout the operation process.

[0096] Specifically, in this embodiment, first, the data is divided into a fixed number of logical partitions according to the access frequency through the normalization formula , and then the current load of each storage node is weighted and scored using the load function , and the scheduling order of data access requests is determined according to the scoring results; subsequently, all the data is divided into three categories: hot, warm, and cold through the classification function , and storage strategies are formulated respectively; for data items with high access frequencies, the caching scheduling timing and cache capacity are determined through the prediction function , and a data migration operation is performed when necessary, and its decision basis is determined by the migration function , thus realizing the dynamic management and scheduling of data between different storage media and breaking through the technical bottlenecks existing in traditional methods when dealing with dynamic data environments.

[0097] Embodiment 2

[0098] The second embodiment of the present invention is different from the previous embodiment in that:

[0099] When the above-mentioned functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), magnetic disks, or optical discs that can store program codes.

[0100] This application is described with reference to the flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to the embodiments of this application. It should be understood that each flow and / or block in the flowchart and / or block diagram, as well as the combination of flows and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing devices generate a device for implementing the functions specified in Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0101] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in this computer-readable memory generate a manufactured article including an instruction device, and this instruction device implements the functions specified in Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0102] These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to generate a computer-implemented process. Thus, the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0103] Although the preferred embodiments of the present application have been described, additional changes and modifications can be made to these embodiments by those skilled in the art once they learn the basic creative concepts. Therefore, the appended claims are intended to be construed to include the preferred embodiments as well as all changes and modifications that fall within the scope of the present application.

[0104] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalent technologies, the present application is also intended to include these modifications and variations.

Claims

1. A method for storing a computing power service data set, characterized in that: including classifying the data to be stored according to the data type and update frequency, and extracting specific features of each data item; determining the data writing order based on the extracted features, historical writing records, and the current system state, and performing the data writing operation in accordance with the order; calculating the hash value for each data item separately after the data writing is completed, and transmitting each data item and its corresponding hash value to a predetermined number of storage nodes for data consistency verification; monitoring the access information of each storage node in real time, dividing all the data into a fixed number of logical partitions according to the collected access frequency, response latency, and resource utilization data, and scheduling the data access requests according to the load conditions of each storage node; dividing the data categories according to the data access frequency and data timeliness, and performing data caching operations on the data; The determination of the data writing order includes Extract the specific attributes that make up the feature vector for each data item , where represents the j attributes of the i-th data item and n is the total number of attributes. Use the pre-set weight vector to calculate the weighted sum of attributes, is the weight corresponding to the j-th attribute, is the weighted sum of the attributes of data item i. Through the formula: ； Obtain the basic score of the data item and introduce the historical write record function It is used to quantify the average write delay or throughput recorded in the previous operations of the i-th data item, and define the current system state quantity S to reflect the overall load cache occupancy rate and response delay of the system at the data write moment, and construct a comprehensive scoring model, where represents the historical write record function of the i-th data item, and its value represents the historical write delay or throughput is the comprehensive score of the i-th data item, through the formula: ； Calculate the comprehensive score of the i-th data item, where and are preset coefficients that respectively measure the influence of data attribute scores, historical records, and the current system state on the writing order. The comprehensive score is used to predict the writing time, that is, let ； After the data writing operation is completed, record the actual writing time of each data item And construct a loss function to measure the difference between the predicted value and the actual value and the violation of the order between data items. The loss function is defined as: ； Among them, represents the write time predicted according to the comprehensive score, is a balance parameter between 0 and 1, used to trade off between prediction error and sequential consistency, and the set is defined as all data item pairs that need to satisfy the predetermined write order constraint, where the order constraint requires that for each pair must satisfy the condition; using the gradient descent method to perform online updates on the weight parameters in the model, and its update formula is: ； wherein, is the updated weight, is the current weight, L is the overall loss function, represents the learning rate, and the update is performed in real time after each data write operation.

2. The method for storing a computing power service data set according to claim 1, characterized in that: The extraction of specific features of each data item includes dividing the data into two distinct categories of high-speed updated data and low-speed updated data according to the data type and update frequency, extracting the specific attributes of each data item for the high-speed updated data and low-speed updated data respectively, and recording the attribute values of each data item.

3. The method for storing a computing power service data set according to claim 1, characterized in that: The separate calculation of the hash value includes calculating the hash value for each data item using a fixed algorithm, appending the calculated hash value to the corresponding data item, and transmitting each data item appended with the hash value to a preset number of storage nodes for subsequent consistency verification.

4. The method for storing a computing power service data set according to claim 3, characterized in that: The data consistency verification includes comparing each received data item and its appended hash value within each storage node, using a distributed consensus algorithm to perform consistency judgment on the hash values of each data item in all storage nodes, performing conflict detection on a data item when it is found that the hash value of the data item does not match, and performing data synchronization operations on the data item according to the primary-backup synchronization rule.

5. The method for storing a computing power service data set according to claim 4, wherein: The real-time monitoring of the access information of each storage node includes that the steps of real-time monitoring of the data access information of each storage node include collecting the data access frequency, response latency, and resource utilization on each storage node, transmitting the collected access information to the central scheduling module, and recording the transmitted information in the central scheduling module according to a preset data collection period; Among them, the logical partitioning of all the data according to the monitoring data recorded in the central scheduling module includes counting the data access frequency, dividing the data into a fixed number of logical partitions according to the statistical results, determining the scheduling order of the data access requests according to the current load conditions of each storage node, and allocating the data access requests to the corresponding storage nodes according to the scheduling order.

6. The method for storing a computing power service data set according to claim 5, wherein: The scheduling of the data access requests includes counting the access records of all data items, dividing all the data into hot data, warm data, and cold data respectively according to the statistical results, storing the divided hot data in a high-speed storage medium, storing the divided cold data in a low-speed storage medium, and performing a predetermined intermediate storage operation on the divided warm data.

7. The system of a computing power service data set storage method as described in claim 6, characterized in that: The execution of the data caching operation includes the step of periodically evaluating the data access status in each storage medium, which includes counting the data access frequencies in each storage medium within a fixed evaluation period, determining a specific data migration scheme between different storage media based on the statistical results, and performing the data migration operation from a low-speed storage medium to a high-speed storage medium, so that the data migration operation and the data caching operation are continuously carried out in the operation process.

8. A computer device, comprising a memory and a processor, the memory storing a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method described in any one of claims 1 to 7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the method described in any one of claims 1 to 7.

Images