Data storage and scheduling method based on intelligent layering and cloud platform system

Abstract

The invention discloses a data storage and scheduling method based on intelligent layering and a cloud platform system, and belongs to the technical field of cloud storage, and the method comprises the steps: collecting to-be-stored data of a cloud platform, extracting multi-dimensional features, formulating a label rule, and generating a unique layering label containing a storage heat value label and a layering storage specification table; constructing a distributed storage node topology through a minimum connection tree algorithm, completing the distribution of a data primary copy and a redundant copy based on a weighted matching degree, and outputting a topological structure, a hierarchical position table and a node operation state; modeling a scheduling problem as a Markov decision process, defining a reward function, solving an optimal strategy through a value iteration algorithm, and generating an intelligent scheduling instruction set; executing the instruction and updating the hierarchical position table, writing a data hash value into a scheduling voucher chain, and feeding back a historical data optimization hierarchical strategy; through intelligent layering and dynamic scheduling, collaborative optimization of storage cost, access performance and reliability is realized, and the method is suitable for a cloud platform large-scale data storage scene.

Description

A data storage and scheduling method and cloud platform system based on intelligent hierarchical structure Technical Field

[0001] This invention belongs to the field of cloud storage technology, specifically a data storage and scheduling method and cloud platform system based on intelligent tiering. Background Technology

[0002] With the rapid development of cloud computing technology, cloud platforms have become the core carrier for enterprises and individuals to store massive amounts of data. The explosive growth of data scale and diversified application needs have placed higher demands on the resource allocation efficiency, data access performance, security and reliability, and cost control of cloud storage systems. However, existing cloud storage technologies still have many shortcomings in practical applications: On the one hand, traditional data tiered storage strategies often rely on single-dimensional characteristics, such as coarse division based on file size or simple access frequency, without fully integrating multi-dimensional attributes such as data security compliance requirements, disaster recovery levels, and retrieval priorities. This results in a lack of targeted tiered labels, making it impossible to accurately match the differentiated storage needs of different data. For example, the high security requirements of core business data and the low-cost storage needs of ordinary office documents are not effectively distinguished, which can easily lead to resource waste of high resources and low demand or performance bottlenecks of low resources and high demand. At the same time, the construction of storage node topology often adopts a fixed connection mode, and node matching only considers some hardware attributes without combining dynamic factors such as network latency, geographical distribution, and load status for global optimization. The replication allocation lacks a scientific basis and it is difficult to balance data integrity and storage resource utilization. On the other hand, existing cloud platform scheduling mechanisms are mostly driven by static rules, lacking the ability to adapt to dynamic changes in system status. When storage node load fluctuates, network link conditions change, or data access patterns change, they cannot adjust data storage location, replica count, and index structure in real time, leading to increased data access latency and decreased system availability. Furthermore, scheduling operations lack traceable security mechanisms, and historical scheduling performance data does not form an effective feedback loop. Layering strategies and scheduling rules are difficult to continuously optimize, and after long-term operation, an imbalance between storage costs, access performance, and reliability is likely to occur, failing to meet the full lifecycle optimization needs of large-scale, diverse cloud data storage. Therefore, there is an urgent need for a cloud platform method that can accurately extract multi-dimensional data features, intelligently match storage resources, dynamically optimize scheduling, and provide full-process traceability to solve problems such as extensive layering, inaccurate matching, rigid scheduling, and lack of closed-loop optimization in existing technologies, thereby improving the overall performance of cloud storage systems. Summary of the Invention

[0003] To address the shortcomings of existing technologies, this invention proposes a data storage and scheduling method and cloud platform system based on intelligent hierarchical structure. The method involves collecting data to be stored from the cloud platform, extracting multi-dimensional features, and formulating labeling rules to generate unique hierarchical labels with storage heat values ​​and a hierarchical storage specification table. A distributed storage node topology is constructed using the minimum connection tree algorithm, and the allocation of primary and redundant data replicas is completed based on weighted matching degree. The topology structure, hierarchical location table, and node operating status are output. The scheduling problem is modeled as a Markov decision process, a reward function is defined, and the optimal strategy is solved using a value iteration algorithm to generate an intelligent scheduling instruction set. Instructions are executed, the hierarchical location table is updated, data hash values ​​are written to the scheduling credential chain, and historical data is fed back to optimize the hierarchical strategy. This invention achieves synergistic optimization of storage cost, access performance, and reliability through intelligent hierarchical structure and dynamic scheduling, making it suitable for large-scale data storage scenarios on cloud platforms.

[0004] To achieve the above objectives, the present invention provides the following technical solution:

[0005] A data storage and scheduling method based on intelligent hierarchical structure, comprising:

[0006] S1: Collect data to be stored on the cloud platform, extract multi-dimensional features of the data, formulate label generation rules according to the preset hierarchical strategy, generate a unique hierarchical label with storage heat value for each data, establish a data-label association mapping relationship, and output the hierarchical dataset and the hierarchical storage specification table corresponding to the label.

[0007] S2: Taking the hierarchical dataset and hierarchical storage specification table as input, the minimum connection tree algorithm is used to construct the distributed storage node topology, allocate data according to the hierarchical specification matching degree, and output the storage topology structure, hierarchical location table and node running status.

[0008] S3: Model the data scheduling problem as a Markov decision process, use the value iteration algorithm to intelligently schedule the data to be stored, and output an intelligent scheduling instruction set;

[0009] S4: Execute the intelligent scheduling instruction set to complete the data scheduling operation, update the hierarchical location table, write the updated storage topology, hierarchical location table and hash value of the intelligent scheduling instruction set into the scheduling credential chain, extract the historical effect data in the scheduling credential chain and feed it back to S1, and optimize the hierarchical strategy and hierarchical storage specification table.

[0010] Specifically, the extraction of multidimensional features from the data includes:

[0011] Obtain the original file information of the data to be stored, and parse out the structured metadata containing file format, file size, creation timestamp, and modification timestamp;

[0012] Based on a preset time window, the historical access logs of the data to be stored within the time window are statistically analyzed to calculate the number of accesses, access time distribution characteristics, and the time interval of the most recent access, forming an access feature vector. Security policies, backup policies, and compliance policies associated with the data to be stored are read from the cloud platform's data management policy library, and a security and compliance feature vector containing its security level identifier, backup policy identifier, and compliance requirement description is output. Historical retrieval records of the data to be stored are queried from the cloud platform's user behavior analysis system, and its click-through rate, average retrieval response time, and weight relationship with frequently associated search terms are calculated to generate a retrieval feature vector.

[0013] The structured metadata, access feature vector, security and compliance feature vector, and retrieval feature vector are fused together to form a multidimensional data feature vector.

[0014] Specifically, the step of formulating tag generation rules based on a preset layering strategy includes:

[0015] Based on the multidimensional feature vectors of the data, a hierarchical decision-making model is constructed, wherein the decision dimensions include access frequency, security level, disaster recovery level, and retrieval priority.

[0016] For the access frequency dimension, the number of accesses in the access feature vector is compared with preset cold, warm, and hot access frequency threshold ranges to determine the access heat range to which it belongs, and the access heat value is calculated based on the determination result, and the access heat value is mapped to the corresponding storage heat value tag; for the security level dimension, the security level identifier is extracted from the security and compliance feature vector, and according to the preset security policy mapping table, the encryption algorithm strength, access control list level, and physical or logical isolation level of the storage area required for data storage are determined based on the security level identifier, and a security level tag is generated; for the disaster recovery level dimension, according to the backup policy identifier in the security and compliance feature vector, according to the preset disaster recovery policy mapping table, the minimum number of data replicas, the geographical distribution requirements of replica storage, and the time target for data recovery are determined, and a disaster recovery level tag is generated; for the retrieval priority dimension, according to the click rate, response time, and associated search term weights in the retrieval feature vector, a weighted scoring algorithm is used to calculate the retrieval priority score, and a retrieval priority tag is generated according to the range in which the retrieval priority score is located.

[0017] The storage heat value tag, security level tag, disaster recovery level tag, and retrieval priority tag are combined according to a preset encoding order to generate the unique hierarchical tag, and recorded in the data-tag association mapping relationship.

[0018] Specifically, the process of constructing the distributed storage node topology includes:

[0019] Collect the physical attributes of all available storage nodes; the physical attributes include node geographical location, total storage capacity and remaining storage capacity, storage media type, network bandwidth, current CPU and memory load rate, and node availability level, forming a node attribute list;

[0020] Based on the node attribute list, the connection weight between any two storage nodes is calculated to form a node connection weight matrix. The calculation of the connection weight comprehensively considers the measured network latency, bandwidth availability ratio, geographical distance, and storage medium performance differences. Among these factors, network latency and geographical distance are negatively correlated with the weight, while bandwidth availability and medium performance are positively correlated with the weight.

[0021] Using all storage nodes as vertices and the node connection weight matrix as edge weights, the minimum connection tree algorithm is applied to construct a tree-like logical topology.

[0022] Based on the generated tree-shaped logical topology, and according to the preset cross-regional disaster recovery requirements, at least one backup node that meets the minimum geographical distance threshold and network quality threshold is dynamically associated with each primary storage node from the node attribute list, and a backup logical connection is established to form a distributed storage node topology.

[0023] Specifically, the process of allocating data based on hierarchical specification matching degree and outputting the storage topology, hierarchical location table, and node running status includes:

[0024] For each data block in the hierarchical dataset, the storage resource specification parameters in the hierarchical storage specification table corresponding to its unique hierarchical label are encapsulated into the storage demand vector of that data block. At the same time, based on the node attribute list, the remaining storage capacity, storage medium type, network bandwidth, availability level, and geographical location information of each storage node are extracted and encapsulated into a storage supply vector.

[0025] A multi-dimensional matching score is calculated between the storage demand vector of each data block and the storage supply vector of each storage node using weighted Euclidean distance. The storage node with the highest matching score is selected as the primary storage node for each data block to write data. Based on the minimum number of replicas required by the disaster recovery level label in the hierarchical label of the data block, the node with the second highest matching score and that meets the disaster recovery geographical distribution requirements is selected from the remaining storage nodes as the redundant replica storage node to complete the replica writing.

[0026] The allocation results are recorded, and a hierarchical location table is generated. This table records the primary storage node identifier, all redundant replica storage node identifiers, specific storage paths, and allocation timestamps for each data block. Simultaneously, based on dynamic updates to the node attribute list, a node operational status table is generated, recording the current used capacity, CPU and memory load rates, network bandwidth utilization, and online status of each storage node in real time. The final output includes the distributed storage node topology, the hierarchical location table, and the node operational status table.

[0027] Specifically, modeling the data scheduling problem as a Markov decision process includes:

[0028] The scheduling state space of the Markov decision process is composed of storage topology, hierarchical location table and node running status. The storage topology describes the logical connection and disaster recovery relationship between nodes, the hierarchical location table records the storage location of the current primary and redundant replicas of each data block, and the node running status characterizes the real-time resource load and availability of each storage node.

[0029] Define a set of scheduling actions for the Markov decision process; the set of scheduling actions includes at least:

[0030] Migrate the specified data block from its current storage node to another target storage node;

[0031] Create a new redundant copy of the specified data block on the target storage node;

[0032] Delete a redundant copy of a specified data block; adjust the index structure of a specified data block in the retrieval system;

[0033] For each action in the set of scheduled actions, a state transition probability is constructed. The state transition probability function quantifies the probability that the system state will transition from the current state to the next state after executing any specific action based on a historical statistical model. The probability mainly depends on three dynamic factors: the probability of changes in storage node resource utilization, the probability of changes in network link status, and the probability of changes in data access patterns.

[0034] Define a comprehensive instant reward function; the comprehensive instant reward function shall contain at least four weighted components:

[0035] A cost-saving incentive component that is inversely proportional to the overall system storage cost;

[0036] A performance improvement bonus component that is inversely proportional to the average data access latency;

[0037] A reliability bonus component that is proportional to the overall availability and data integrity of the system;

[0038] An operational overhead penalty component that is proportional to the network bandwidth and computing resources consumed in performing the action.

[0039] Specifically, the value iteration algorithm is used for intelligent scheduling of the data to be stored, including:

[0040] The value iteration algorithm is applied to solve the Markov decision process. Specifically, starting from the initial assumed state value, the value function estimate of each state in the entire scheduling state space is iteratively updated using the defined state transition probability function and the comprehensive immediate reward function. When the value function updates of all states tend to stabilize, that is, convergence, the scheduling action that can maximize its long-term cumulative reward is selected for each state, and a set of optimal scheduling strategies is output.

[0041] The optimal scheduling strategy is transformed into executable operation instructions based on the current scheduling state space, and finally an intelligent scheduling instruction set is output; the operation instructions define specific data migration operations, copy operations or index update operations.

[0042] Specifically, the execution of the intelligent scheduling instruction set to complete the data scheduling operation includes:

[0043] Read the input intelligent scheduling instruction set, parse the operation type, target data block identifier, source / target storage node identifier and other parameters of each instruction, sort all instructions according to the preset execution priority of instruction type and the potential data dependency relationship between instructions, and generate an executable instruction queue;

[0044] The scheduler starts from the head of the generated executable instruction queue and sequentially retrieves and executes each instruction.

[0045] When the instruction is for data migration, the scheduler first looks up the current primary storage node identifier of the specified data block in the hierarchical location table. Then, the scheduler reads the specified data block from the source storage node and transmits it to the target storage node via the network. After the target storage node completes the data writing and verification, the scheduler updates the hierarchical location table, changes the primary storage node identifier of the data block to the target storage node, and removes the original location record.

[0046] When the instruction is to create a replica, the scheduler creates a complete replica of the data block on the target storage node specified by the instruction. After the replica is created and verified to be successful, the scheduler updates the hierarchical location table and adds the identifier of the target storage node to the list of redundant replica storage nodes for the data block.

[0047] When the instruction is for index update, the scheduler searches for and updates all index entries related to the data block in the distributed retrieval index according to the data block identifier specified in the instruction, ensuring that the storage location and other attributes it points to are consistent with the records in the current hierarchical location table;

[0048] After all instructions have been executed, the scheduler updates the system status record synchronously based on the instruction execution results, and at the same time, triggers the sampling update of the node running status.

[0049] Specifically, writing the updated storage topology, hierarchical location table, and hash values ​​of the intelligent scheduling instruction set into the scheduling credential chain includes:

[0050] Extract the core graph theory representation of the updated storage topology generated after the scheduler completes the data scheduling operation, namely the list of storage nodes and the list of node connection relationships. Extract the complete record of the hierarchical location containing all data block identifiers, primary storage nodes, redundant replica storage nodes and storage paths from the updated hierarchical location table. Then extract the instruction serialization byte stream from the intelligent scheduling instruction set on which this execution is based.

[0051] The extracted list of storage nodes, list of node connections, complete record of hierarchical positions, and instruction serialization byte stream are concatenated according to a predefined format to form a scheduling context information block. An encrypted hash function is then applied to the scheduling context information block to generate a unique scheduling operation digest hash value of fixed length.

[0052] Obtain the execution completion time of this scheduling operation, generate a scheduling operation timestamp, obtain its hash value from the current end block of the scheduling credential chain as the hash of the preceding block of the system state before this scheduling operation, and combine the scheduling operation digest hash value, scheduling operation timestamp and preceding block hash to form a new block structure to be added to the chain.

[0053] The constructed block structure is linked to the end of the existing scheduling credential chain, making it the latest block on the chain, thus completing the writing and updating of the scheduling credential chain.

[0054] A cloud platform system based on intelligent layering includes: a layered label generation module, a data allocation module, an instruction generation module, and a scheduling and execution module;

[0055] The layered label generation module is used to collect multi-dimensional information of the data to be stored on the cloud platform, and generate unique layered labels and layered storage specification tables through feature fusion and layered rule design.

[0056] The data allocation module is used to construct a distributed storage node topology. Based on the precise matching of data storage requirements and node resource supply, it completes the reasonable allocation of primary and redundant data replicas and outputs the storage topology structure, hierarchical location table, and node operating status.

[0057] The instruction generation module is used to model the data scheduling problem as a Markov decision process, use the value iteration algorithm to intelligently schedule the data to be stored, and generate an intelligent scheduling instruction set.

[0058] The scheduling execution module is used to execute intelligent scheduling instructions to complete data scheduling operations, update the hierarchical location table, write the updated storage topology, hierarchical location table and hash value of intelligent scheduling instruction set into the scheduling credential chain, extract historical effect data from the scheduling credential chain and feed it back to the hierarchical tag generation module to optimize the hierarchical strategy and hierarchical storage specification table.

[0059] Compared with the prior art, the beneficial effects of the present invention are:

[0060] This invention proposes a data storage and scheduling method based on intelligent hierarchical structure. This method achieves precise matching between data storage needs and storage resources through multi-dimensional feature extraction and refined hierarchical label design. Hierarchical labels generated based on multiple decision dimensions such as access frequency and security level can specifically meet the storage security, disaster recovery, and retrieval needs of different data. Combined with a distributed storage topology constructed using the minimum connection tree algorithm and a weighted matching degree allocation mechanism, it can both ensure data integrity and availability through cross-regional backup node associations and optimize storage node resource utilization, avoiding redundancy and waste.

[0061] This invention proposes a data storage and scheduling method based on intelligent tiering. Relying on the intelligent scheduling mechanism of Markov decision processes and closed-loop optimization design, it enhances the system's dynamic adaptability and overall performance. The optimal scheduling strategy, solved through a value iteration algorithm, dynamically balances storage costs, access performance, reliability, and operational overhead, responding in real-time to changes in node load, network conditions, and data access patterns, reducing access latency and resource consumption. Furthermore, the introduction of a scheduling credential chain not only ensures the traceability and security of scheduling operations, but also allows feedback from historical performance data to continuously optimize the tiering strategy, enabling performance iteration and upgrades of the storage system throughout its entire lifecycle, achieving optimal synergy between cost, performance, and reliability. Attached Figure Description

[0062] Figure 1 is a schematic diagram of a data storage and scheduling method based on intelligent hierarchical structure according to the present invention;

[0063] Figure 2 is a flowchart illustrating the principle of a data storage and scheduling method based on intelligent hierarchical structure according to the present invention.

[0064] Figure 3 is a system architecture diagram of a cloud platform based on intelligent layering according to the present invention. Detailed Implementation

[0065] Example 1:

[0066] Referring to Figures 1 and 2, one embodiment of the present invention provides a data storage and scheduling method based on intelligent hierarchical structure, comprising the following steps:

[0067] S1: Collect data to be stored on the cloud platform, extract multi-dimensional features of the data, formulate label generation rules according to the preset hierarchical strategy, generate a unique hierarchical label with storage heat value for each data, establish a data-label association mapping relationship, and output the hierarchical dataset and the hierarchical storage specification table corresponding to the label.

[0068] S2: Taking the hierarchical dataset and hierarchical storage specification table as input, the minimum connection tree algorithm is used to construct the distributed storage node topology, allocate data according to the hierarchical specification matching degree, and output the storage topology structure, hierarchical location table and node running status.

[0069] S3: Model the data scheduling problem as a Markov decision process, use the value iteration algorithm to intelligently schedule the data to be stored, and output an intelligent scheduling instruction set;

[0070] S4: Execute the intelligent scheduling instruction set to complete the data scheduling operation, update the hierarchical location table, write the updated storage topology, hierarchical location table and hash value of the intelligent scheduling instruction set into the scheduling credential chain, extract the historical effect data in the scheduling credential chain and feed it back to S1, and optimize the hierarchical strategy and hierarchical storage specification table.

[0071] In summary, this embodiment first collects multidimensional features of the data to be stored, and generates hierarchical labels and specification tables containing storage heat value markers based on decision dimensions such as access frequency and security level; then, it constructs a distributed storage topology using the minimum connection tree algorithm and completes the allocation of primary and secondary data replicas according to weighted matching degree; subsequently, it models the scheduling problem as a Markov decision process, and generates the optimal scheduling instruction that balances cost, performance, and reliability through a value iteration algorithm; finally, it executes the instruction and updates the system state, writes key data into the scheduling credential chain, and continuously optimizes the hierarchical strategy by providing feedback on historical performance data, thereby achieving dynamic adaptation and performance upgrade of the storage system.

[0072] For example, suppose a sales employee of a small e-commerce company uploads a 5MB Excel-formatted weekly report to the company's cloud drive. The input is the 5MB Excel-formatted weekly report file, along with three types of auxiliary data from the cloud platform: first, the original file information, including at least the creation time (2024-06-30) and modification time (2024-06-30); second, the cloud platform's policy library, namely the security and backup policies for internal documents of the sales department to which the weekly report belongs; and third, the user behavior analysis system, such as access and retrieval records of this type of weekly report in the past 30 days. The system automatically parses the structured metadata of the weekly report; analyzes the access logs of the past 30 days, finding that the sales manager views it twice a week and the finance department checks it once a week, with access concentrated on Monday mornings and the most recent access 3 days apart, generating an access feature vector; reads the internal confidentiality and security policy from the policy library, namely, AES-128 encryption is required, and access is only available to the sales and finance departments; local dual-copy backup policy, generating a security and compliance feature vector; queries the retrieval records, finding that the weekly report was retrieved 3 times in the past 30 days, with an average response time of 0.8 seconds, and the weight of the search terms associated with the 2024 sales data is 0.6, generating a retrieval feature vector, and finally merges them into a complete multi-dimensional feature vector.

[0073] The system determines the tags for each dimension according to preset rules. For example, in terms of access frequency, 6 accesses per month are considered hot data, and the mapped storage hot value tag is Hot-08; in terms of security level, internal confidentiality corresponds to AES-128 encryption and department-level access control, generating a Security-L2 tag; in terms of disaster recovery level, local dual replicas correspond to 2 replicas and storage in the same data center, generating a Disaster Recovery-B2 tag; in terms of retrieval priority, a weighted score of 0.72 generates a Retrieval-P1 tag, which is combined into a unique label according to the encoding order: Hot-08_Security-L2_Disaster Recovery-B2_Retrieval-P1. At the same time, a tiered storage specification table is generated, which clearly requires: SSD storage media, AES-128 encryption, 2 replicas, and department-level access control.

[0074] Then, the attributes of three available storage nodes are collected. For example, node A: data center 1, SSD media, remaining capacity 100GB, bandwidth 1Gbps, CPU load 30%; node B: data center 1, SSD media, remaining capacity 80GB, bandwidth 1Gbps, CPU load 40%; node C: data center 2, HDD media, remaining capacity 500GB, bandwidth 500Mbps. The node connection weights are calculated as follows: A and B are in the same data center and have low latency, with a weight of 0.9; A and C are in different data centers, with a weight of 0.6.

[0075] A tree topology centered on node A and node B is constructed using the Kruskal algorithm, with B associated with A as a backup node. Then, the storage demand vector (SSD, encryption, dual replicas) from the weekly report is matched with the node supply vector. The weighted Euclidean distance calculation yields a matching degree of 0.95 for node A, 0.92 for node B, and 0.6 for node C. Finally, A is determined to be the primary storage node and B to be the replica node, completing the data writing process.

[0076] The final output includes three parts: First, the weekly report is successfully stored on node A (primary replica) and node B (redundant replica), along with their storage paths; second, the hierarchical location table records the primary and secondary node identifiers, storage paths, and allocation timestamps; and third, the node running status table is updated to: node A has 95GB of remaining capacity, node B has 75GB of remaining capacity, and CPU and bandwidth loads are normal.

[0077] The extracted multidimensional features of the data specifically include:

[0078] A1: Obtain the original file information of the data to be stored, and parse out the structured metadata containing file format, file size, creation timestamp and modification timestamp;

[0079] A2: Based on a preset time window, statistically analyze the historical access logs of the data to be stored within the time window, calculate the number of accesses, access time distribution characteristics, and the time interval of the most recent access, and form an access feature vector; read the security policy, backup policy, and compliance policy associated with the data to be stored from the data management policy library of the cloud platform, and output a security and compliance feature vector containing its security level identifier, backup policy identifier, and compliance requirement description; query the historical retrieval records of the data to be stored from the user behavior analysis system of the cloud platform, calculate its click-through rate, average retrieval response time, and its weight relationship with high-frequency associated search terms, and generate a retrieval feature vector;

[0080] Furthermore, the process of generating the retrieval feature vector includes:

[0081] First, based on the globally unique identifier of the data to be stored, all historical search records related to the data to be stored are extracted from the user behavior analysis system database of the cloud platform. Each search record contains search keywords, search timestamp, search result list and user click behavior identifier in the result list. The system sorts these records in ascending order by timestamp to generate an ordered search behavior sequence of the data to be stored.

[0082] Second, based on the generated ordered retrieval behavior sequence, the system calculates two core basic features, including: First, the retrieval click-through rate, which is calculated by counting the total number of times the data to be stored is returned as a retrieval result in the sequence and the total number of times the user actually clicks on the data to be stored in these return events, and dividing the total number of clicks by the total number of returns to obtain the click-through rate value; Second, the average retrieval response time, which is calculated by filtering out the records in the sequence where the data to be stored was successfully retrieved and returned, extracting the response time consumed by the system from receiving the query to preparing to return the data for each record, summing the response times of all valid records and dividing by the total number of records to obtain the average retrieval response time value.

[0083] Third, the system analyzes the search keywords of each record in the ordered search behavior sequence, obtains a standardized search term set by word segmentation and removal of stop words, counts the frequency of each standardized search term in the entire sequence, and filters out terms with frequencies higher than a preset threshold to form a high-frequency related search term set for the data. Then, a weight is calculated for each related search term in the high-frequency related search term set. This weight is not only based on its frequency of occurrence, but also combined with the density of the time distribution of the search and the importance of the user role that initiated the search. Finally, a quantified weight value is generated for each related search term.

[0084] Fourth, the calculated search click-through rate and average search response time are combined and packaged with the obtained high-frequency related search term set and its corresponding quantitative weight values ​​according to a preset fixed format. Specifically, the first two dimensions of the packaged search feature vector are fixed as search click-through rate and average search response time, respectively, while the remaining dimensions are filled with the weight values ​​of each high-frequency related search term in sequence. If the number of high-frequency related search terms is less than the preset dimensions, it is filled with zero values. Finally, the system outputs this structured search feature vector.

[0085] A3: The structured metadata, access feature vector, security and compliance feature vector, and retrieval feature vector are fused together to form a multidimensional data feature vector.

[0086] The step of formulating tag generation rules based on a preset layering strategy includes:

[0087] S1.1: Based on the multidimensional feature vector of the data, construct a hierarchical decision model, wherein the decision dimensions include access frequency, security level, disaster recovery level and retrieval priority;

[0088] Furthermore, the specific steps of S1.1 include:

[0089] First, using multi-dimensional feature vectors of data as input, a hierarchical decision-making model is established, comprising four core dimensions: access frequency, security level, disaster recovery level, and retrieval priority. Sub-features corresponding to each dimension are extracted from the multi-dimensional feature vectors of data, including: extracting the total number of accesses per unit time window, peak access frequency, and recent access time interval from the access feature vector to form an access frequency feature set; extracting data encryption level, access control list complexity, and storage isolation requirements from the security and compliance feature vector to form a security level feature set; extracting the minimum number of data replicas, replica geographical distribution constraints, and the target data recovery time from the same feature vector to form a disaster recovery level feature set; and extracting average click-through rate, high percentile response time, and the weights of the top N high-frequency related search terms from the retrieval feature vector to form a retrieval priority feature set.

[0090] Second, the extracted feature sets of each dimension are independently quantified and scored, including: for the access frequency feature set, a time decay weighted algorithm is applied to calculate the popularity score, and a standardized access frequency score is generated through piecewise function mapping; for the security level feature set, a pre-defined security policy mapping matrix is ​​used to map the encryption level, access control complexity, and isolation requirements into a standardized security level score; for the disaster recovery level feature set, a pre-defined disaster recovery policy scoring card is used to calculate a standardized disaster recovery level score by weighting the number of replicas, geographical distribution constraints, and recovery time targets; for the retrieval priority feature set, a pre-defined weighted comprehensive calculation model is used to proportionally integrate the average click-through rate, high percentile response time, and high-frequency related keyword weights into a standardized retrieval priority score. All dimension scores are normalized to the range of 0 to 1.

[0091] Third, the obtained four standardized dimension scores are input into the hierarchical decision fusion unit. The hierarchical decision fusion unit first performs a weighted summation of the scores of each dimension according to the preset dimension priority weights to calculate a preliminary comprehensive hierarchical score. Then, this comprehensive score and the independent scores of each dimension are input into a preset hierarchical judgment rule set. The hierarchical judgment rule set compares the access frequency score with the three-level thresholds of hot, warm, and cold, the security level score with the three-level thresholds of high, medium, and low, the disaster recovery level score with the three-level thresholds of critical, important, and general, and the retrieval priority score with the three-level thresholds of high, medium, and low. The fusion unit integrates the independent comparison results and the comprehensive score, and based on the preset decision matrix, finally generates a clear hierarchical level identifier for the data object.

[0092] S1.2: For the access frequency dimension, the number of accesses in the access feature vector is compared with the preset cold, warm and hot access frequency threshold ranges to determine the access heat range to which it belongs, and the access heat value is calculated based on the determination result, and the access heat value is mapped to the corresponding storage heat value mark.

[0093] Furthermore, the threshold ranges for cold, warm, and hot access frequencies used to determine the access frequency dimension in the hierarchical decision-making model are not fixed, but have adaptive adjustment capabilities:

[0094] (1) The system continuously monitors the historical scheduling effect recorded in the scheduling credential chain, especially the change in average access delay after data with different access frequencies are scheduled;

[0095] (2) When it is found that data in any threshold range, such as the warm data range, is frequently mis-scheduled, such as data that should be scheduled to high-speed storage but is left in low-speed storage due to unreasonable threshold, resulting in a significantly higher average access latency than expected, the threshold optimization process is triggered.

[0096] (3) The threshold optimization process adopts a reinforcement learning framework, taking the current threshold range as the state, the small adjustment of the threshold as the action, and the reduction of the overall data access latency as the reward signal. The reinforcement learning framework is the prior art in this field and is not an inventive solution of this application, so it will not be described in detail here.

[0097] (4) By using the policy gradient algorithm, the threshold parameters are iteratively optimized so that the threshold range can dynamically adapt to changes in data access patterns, thereby improving the long-term effectiveness of the hierarchical strategy.

[0098] The policy gradient algorithm is existing technology in this field and is not an inventive solution of this application, so it will not be described in detail here.

[0099] S1.3: For the security level dimension, extract the security level identifier from the security and compliance feature vector, and based on the preset security policy mapping table, determine the encryption algorithm strength, access control list level, and physical or logical isolation level of the storage area required for data storage according to the security level identifier, and generate a security level tag.

[0100] Furthermore, the specific steps in S1.3 include:

[0101] First, receive security and compliance feature vectors from the multidimensional feature vectors of the data, parse out the predefined security level identifier from the security and compliance feature vectors, and perform a precise query in the predefined security policy mapping table based on the parsed security level identifier to locate the complete policy entry corresponding to the identifier.

[0102] Second, three core storage security control parameters are extracted from the security policy mapping table entries obtained from the query. The first parameter is the data encryption algorithm strength, which is specifically defined by the encryption algorithm type and key length requirements specified in the mapping table entry. The second parameter is the access control list level, which is specifically defined by the authorization model granularity and audit log requirements specified in the mapping table entry. The third parameter is the storage isolation level, which is specifically defined by the degree of physical or logical isolation required for data storage specified in the mapping table entry.

[0103] Third, the extracted data encryption algorithm strength, access control list level, and storage isolation level are sequentially combined and concatenated according to a preset unified string encoding standard to finally generate a structured security level label that represents all the technical requirements of the data in the security dimension.

[0104] S1.4: For the disaster recovery level dimension, based on the backup policy identifier in the security and compliance feature vector, and according to the preset disaster recovery policy mapping table, determine the minimum number of data replicas, the geographical distribution requirements for replica storage, and the time target for data recovery, and generate a disaster recovery level label.

[0105] Furthermore, the specific steps in S1.4 include:

[0106] First, it receives input from security and compliance feature vectors, parses out predefined backup policy identifiers, and performs a precise query in the preset disaster recovery policy mapping table based on the parsed backup policy identifiers to locate the corresponding complete policy entry.

[0107] Second, from the disaster recovery strategy mapping table entries obtained from the query, three core disaster recovery parameters are extracted. The first parameter is the minimum number of data replicas, which defines the minimum total number of redundant replicas that must be maintained. The second parameter is the geographical distribution requirements of replicas, which defines the minimum geographical interval and deployment location strategy that replica storage must meet. The third parameter is the data recovery time target, which defines the maximum time allowed from the occurrence of a failure to the availability of data recovery and the maximum data loss time window that the system can tolerate.

[0108] Third, the three parameters—minimum number of extracted data copies, geographical distribution requirements of copies, and data recovery time target—are sequentially combined and concatenated according to a preset standardized string encoding format to generate a structured disaster recovery level label.

[0109] S1.5: For the retrieval priority dimension, based on the click-through rate, response time and associated search term weights in the retrieval feature vector, a retrieval priority score is calculated using a weighted scoring algorithm, and a retrieval priority label is generated based on the range in which the retrieval priority score falls. The weighted scoring algorithm is existing technology in this field and is not an inventive solution of this application, so it will not be described in detail here.

[0110] S1.6: Combine the storage heat value tag, security level tag, disaster recovery level tag, and retrieval priority tag according to the preset encoding order to generate the unique hierarchical tag, and record it in the data-tag association mapping relationship.

[0111] The process of constructing the distributed storage node topology includes:

[0112] S2.1: Collect the physical attributes of all available storage nodes; the physical attributes include node geographical location, total storage capacity and remaining storage capacity, storage media type, network bandwidth, current CPU and memory load rate, and node availability level, forming a node attribute list;

[0113] S2.2: Based on the node attribute list, calculate the connection weight between any two storage nodes to form a node connection weight matrix; the calculation of the connection weight comprehensively considers the measured network latency, bandwidth availability ratio, geographical distance and storage medium performance differences, wherein network latency and geographical distance are negatively correlated with the weight, while bandwidth availability and medium performance are positively correlated with the weight.

[0114] Furthermore, the specific steps of S2.2 include:

[0115] First, using the node attribute list as input, for any two storage nodes, obtain their basic metric data, including the measured value of network latency between nodes obtained through network probing, the bandwidth availability ratio between nodes obtained through link monitoring, the geographical distance between nodes calculated through geographical coordinates, and the performance difference value of the storage media of the two nodes obtained by comparing the attribute list.

[0116] Second, the acquired basic metrics are standardized. Specifically, for measured network latency and geographical distance, a standardization function that is negatively correlated with connection quality is used to map larger original values ​​to smaller standardized scores. For bandwidth availability ratio and storage media performance difference, a standardization function that is positively correlated with connection quality is used to map larger original values ​​to larger standardized scores. All standardized scores are mapped to a closed interval between zero and one.

[0117] Furthermore, a standardization function is a mathematical function that transforms data into a function with the same mean and standard deviation. By calculating the z-score, the data is transformed into a standard normal distribution, enabling the comparison and weighting of indicators of different units or magnitudes. The specific calculation process is a conventional method that can be understood and implemented by those skilled in the art, and this application is not limited to a specific partitioning method.

[0118] Third, the standardized scores obtained from the processing are weighted and summed according to the preset global weight allocation scheme to calculate the comprehensive connection weight between each pair of nodes. Among them, the weight coefficients of the standardized scores of network latency and geographical distance are negative, while the weight coefficients of the standardized scores of bandwidth availability ratio and storage medium performance difference are positive. This calculation is repeated for all node pairs in the node attribute list. The final connection weight between each pair of nodes is then filled into the corresponding positions of a two-dimensional matrix with the node identifier as the row and column index, thereby generating a complete node connection weight matrix.

[0119] S2.3: Using all storage nodes as vertices and the node connection weight matrix as edge weights, a tree-shaped logical topology structure is constructed by applying the minimum connection tree algorithm. The minimum connection tree algorithm is existing technology in this field and is not an inventive solution of this application, so it will not be described in detail here.

[0120] S2.4: Based on the generated tree-shaped logical topology, according to the preset cross-regional disaster recovery requirements, dynamically associate at least one backup node that meets the minimum geographical distance threshold and network quality threshold with each primary storage node from the node attribute list, establish backup logical connections, and form a distributed storage node topology.

[0121] Furthermore, the specific steps in S2.4 include:

[0122] First, it receives a tree-shaped logical topology, a list of node attributes, a node connection weight matrix, and preset cross-regional disaster recovery requirements as input. It then parses the disaster recovery requirements and obtains two core constraints: a minimum geographical distance threshold and a network quality threshold. For each main storage node in the topology, it selects nodes whose geographical distance is greater than or equal to the minimum geographical distance threshold from all non-local nodes based on the node attribute list, thus forming a pool of geographically qualified candidate nodes.

[0123] Second, for the geographically qualified candidate node pool generated for each primary storage node, the system queries the node connection weight matrix to obtain the connection weight between the primary node and each candidate node in the pool, and filters out candidate nodes whose connection weight is greater than or equal to the network quality threshold, thus forming a final candidate backup node list that simultaneously satisfies the dual constraints of geographical isolation and network quality.

[0124] Third, based on the preset optimization selection strategy, at least one node is selected as a backup node from the final candidate backup node list for each primary storage node. On the basis of the tree logical topology, a backup logical connection from the primary node to the backup node is added for each pair of primary and backup nodes. After all primary nodes have completed the addition of backup connections, the final distributed storage node topology containing the trunk connection and all backup logical connections is output.

[0125] The process of allocating data based on hierarchical specification matching degree and outputting the storage topology, hierarchical location table, and node running status includes:

[0126] S2.5: For each data block in the hierarchical dataset, encapsulate the storage resource specification parameters in the hierarchical storage specification table corresponding to its unique hierarchical label into the storage demand vector of the data block. At the same time, based on the node attribute list, extract the remaining storage capacity, storage medium type, network bandwidth, availability level, and geographical location information of each storage node, and encapsulate them into a storage supply vector.

[0127] S2.6: The multi-dimensional matching score between the storage demand vector of each data block and the storage supply vector of each storage node is calculated using weighted Euclidean distance. The storage node with the highest matching score is selected as the primary storage node for each data block to write data. Based on the minimum number of replicas required by the disaster recovery level label in the hierarchical label of the data block, the node with the second highest matching score and that meets the disaster recovery geographical distribution requirements is selected from the remaining storage nodes as the redundant replica storage node to complete the replica writing. The weighted Euclidean distance calculation formula is the prior art in this field and is not an inventive solution of this application, so it will not be described in detail here.

[0128] S2.7: Record the allocation results and generate a hierarchical location table. This table records the primary storage node identifier, all redundant replica storage node identifiers, specific storage paths, and allocation timestamps for each data block. Simultaneously, based on the dynamic updates of the node attribute list, generate a node running status table, recording the current used capacity, CPU and memory load rates, network bandwidth utilization, and online status of each storage node in real time. The final output includes the distributed storage node topology, the hierarchical location table, and the node running status table.

[0129] The process of modeling the data scheduling problem as a Markov decision process includes:

[0130] S3.1: The scheduling state space of the Markov decision process is composed of storage topology, hierarchical location table and node running status. The storage topology describes the logical connection and disaster recovery relationship between nodes, the hierarchical location table records the storage location of the current primary and redundant replicas of each data block, and the node running status represents the real-time resource load and availability of each storage node.

[0131] Furthermore, the Markov decision process is a mathematical framework used to model the interaction between an intelligent agent and its environment. Its specific implementation process is existing technology in this field and is not an inventive solution of this application, so it will not be described in detail here.

[0132] S3.2: Define a set of scheduling actions for the Markov decision process; the set of scheduling actions includes at least:

[0133] Migrate the specified data block from its current storage node to another target storage node;

[0134] Create a new redundant copy of the specified data block on the target storage node;

[0135] Delete a redundant copy of a specified data block; adjust the index structure of a specified data block in the retrieval system;

[0136] S3.3: For each action in the set of scheduled actions, construct a state transition probability; the state transition probability function quantifies the probability that the system state will transition from the current state to the next state after executing any specific action based on a historical statistical model; the probability mainly depends on three dynamic factors: the probability of changes in storage node resource utilization, the probability of changes in network link status, and the probability of changes in data access patterns.

[0137] Furthermore, the specific steps of S3.3 include:

[0138] First, a series of state transition records are extracted from historical scheduling logs. Each record contains the system state before the action is executed, the action executed, and the next system state observed after execution. The system state is analyzed to extract features of three dynamic factors: based on the time-series data of resource monitoring of each node before and after the action is executed, quantitative features of changes in storage node resource utilization are extracted; based on the time-series data of network performance between related nodes before and after the action is executed, quantitative features of changes in network link status are extracted; based on the time-series data of related data block access logs before and after the action is executed, quantitative features of changes in data access patterns are extracted; and characteristic historical state transition samples are output.

[0139] Second, using the obtained characteristic historical state transition samples, a state transition model is constructed for each action in the scheduling action set. The state transition model decomposes the probability distribution of the next system state into a joint of three conditionally independent sub-probabilities. Specifically, three conditional probability models are established: the first is a probability model of changes in storage node resource utilization given the current state and the action being executed; the second is a probability model of changes in the status of related network links given the current state and the action being executed; and the third is a probability model of changes in related data access patterns given the current state and the action being executed. Each model is trained using machine learning methods, which are existing technologies in this field and are not the inventive solution of this application, and will not be elaborated here.

[0140] Third, the three conditional probability models trained for each scheduling action are integrated and encapsulated. Then, the system generates a unified callable function for each action, namely the state transition probability function. The state transition probability function receives the current system state and the action to be executed as input, internally calls the corresponding three conditional probability models, multiplies the conditional probability values ​​output by each model, and calculates the final joint probability value of the system state transition after the action is executed.

[0141] S3.4: Define a comprehensive instant reward function; the comprehensive instant reward function shall contain at least four weighted components:

[0142] A cost-saving incentive component that is inversely proportional to the overall system storage cost;

[0143] A performance improvement bonus component that is inversely proportional to the average data access latency;

[0144] A reliability bonus component that is proportional to the overall availability and data integrity of the system;

[0145] An operational overhead penalty component that is proportional to the network bandwidth and computing resources consumed in performing the action;

[0146] S3.5: The value iteration algorithm is applied to solve the Markov decision process. Specifically, starting from the initial assumed state value, the value function estimate of each state in the entire scheduling state space is iteratively updated using the defined state transition probability function and the comprehensive immediate reward function. When the value function updates of all states tend to stabilize, i.e. convergence, the scheduling action that maximizes the long-term cumulative reward is selected for each state, and a set of optimal scheduling strategies is output. The specific calculation process of the value iteration algorithm is the prior art in this field and is not an inventive solution of this application, so it will not be described in detail here.

[0147] This embodiment models the data scheduling problem as a Markov decision process, constructing a complete scheduling state space that includes storage topology, hierarchical location table, and node operating status. This is combined with a multi-faceted action set encompassing data migration, replication operations, and index updates, along with a weighted reward function that integrates cost savings, performance improvements, reliability assurance, and operational overhead penalties, forming an adaptive intelligent scheduling mechanism. Furthermore, a value iteration algorithm iteratively solves for the globally optimal scheduling strategy, enabling precise responses to dynamic scenarios such as storage node load fluctuations, network link quality changes, and data access pattern migrations. It dynamically balances the core contradictions of storage cost versus hardware investment, data access performance versus resource consumption, and system reliability versus operational overhead, reducing data access latency and minimizing ineffective network bandwidth and computing resource consumption. Simultaneously, relying on closed-loop optimization design, it achieves continuous strategy iteration, improving the system's dynamic adaptability and overall operational efficiency in complex cloud environments.

[0148] S3.6: The optimal scheduling strategy is transformed into executable operation instructions based on the current scheduling state space, and finally the intelligent scheduling instruction set is output; the operation instructions define specific data migration operations, copy operations or index update operations.

[0149] Furthermore, the specific steps in S3.6 include:

[0150] First, it receives the optimal scheduling policy and the current scheduling state space as input, parses the current scheduling state space, accurately matches the state features required by the optimal scheduling policy, and obtains the set of optimal scheduling actions specified by the policy that should be executed in the current state.

[0151] Second, for each action in the obtained optimal scheduling action set, translation and instantiation are performed to generate specific executable operation instructions. Specifically, for data migration actions, the hierarchical location table is queried to determine the current source storage node of the data block, and the target storage node is determined by combining the node running status table, generating a migration instruction that includes the operation type, data block identifier, source node identifier, and target node identifier. For replica operation actions, the type is determined as creation or deletion, and the data block identifier, target storage node identifier, and replica identifier are determined, generating the corresponding replica operation instruction. For index update actions, the data block identifier and the list of index fields to be updated are determined, generating the index update instruction.

[0152] Third, all generated specific operation instructions are serialized and logically verified to ensure no resource conflicts or execution loops. Then, the instructions are sorted and grouped according to preset priorities and batch strategies, and finally encapsulated into a complete intelligent scheduling instruction set with metadata identification and output.

[0153] The execution of the intelligent scheduling instruction set to complete the data scheduling operation specifically includes:

[0154] S4.1: Read the input intelligent scheduling instruction set, parse the operation type, target data block identifier, source / target storage node identifier and other parameters of each instruction, sort all instructions according to the preset execution priority of instruction type and the potential data dependency relationship between instructions, and generate an executable instruction queue;

[0155] S4.2: The scheduler starts from the head of the generated executable instruction queue and sequentially retrieves and executes each instruction.

[0156] When the instruction is for data migration, the scheduler first looks up the current primary storage node identifier of the specified data block in the hierarchical location table. Then, the scheduler reads the specified data block from the source storage node and transmits it to the target storage node via the network. After the target storage node completes the data writing and verification, the scheduler updates the hierarchical location table, changes the primary storage node identifier of the data block to the target storage node, and removes the original location record.

[0157] When the instruction is to create a replica, the scheduler creates a complete replica of the data block on the target storage node specified by the instruction. After the replica is created and verified to be successful, the scheduler updates the hierarchical location table and adds the identifier of the target storage node to the list of redundant replica storage nodes for the data block.

[0158] When the instruction is for index update, the scheduler searches for and updates all index entries related to the data block in the distributed retrieval index according to the data block identifier specified in the instruction, ensuring that the storage location and other attributes it points to are consistent with the records in the current hierarchical location table;

[0159] S4.3: After all instructions have been executed, the scheduler updates the system status record synchronously based on the instruction execution results, and at the same time, triggers the sampling update of the node running status.

[0160] The step of writing the updated storage topology, hierarchical location table, and hash values ​​of the intelligent scheduling instruction set into the scheduling credential chain specifically includes:

[0161] B1: Extract the core graph theory representation, namely the list of storage nodes and the list of node connections, from the updated storage topology generated after the scheduler completes the data scheduling operation. Extract the complete record of the hierarchical location, which includes all data block identifiers, primary storage nodes, redundant replica storage nodes and storage paths, from the updated hierarchical location table. Then extract the instruction serialization byte stream from the intelligent scheduling instruction set on which this execution is based.

[0162] Furthermore, the specific steps for B1 include:

[0163] First, the updated storage topology is received as input, all vertices are traversed, the unique identifier and key attributes of each storage node are extracted, and a list of storage nodes is generated. At the same time, all edges are traversed, and the starting node identifier, ending node identifier and connection type attribute of each connection are extracted to generate a list of node connection relationships.

[0164] Second, the updated hierarchical location table is received as input, and a full scan of the table is performed. For each record in the table, its data block identifier, primary storage node identifier, list of identifiers of all redundant replica storage nodes, and precise storage path on each storage node are extracted in sequence. All extracted information is organized according to a preset standard format to generate a complete record of hierarchical location.

[0165] Third, the system receives the intelligent scheduling instruction set to be executed as input, calls the preset serialization engine, serializes the intelligent scheduling instruction set object, and converts all the instructions, parameters, order and metadata contained therein according to the predetermined binary encoding rules to generate a linear instruction serialization byte stream.

[0166] Fourth, the generated list of storage nodes, list of node connection relationships, complete record of hierarchical positions, and instruction serialization byte stream are encapsulated and output together as complete credential source data for constructing scheduling credentials.

[0167] B2: The extracted list of storage nodes, list of node connection relationships, complete record of hierarchical positions, and instruction serialization byte stream are concatenated according to a predefined format to form a scheduling context information block. An encrypted hash function is then applied to the scheduling context information block to generate a unique scheduling operation digest hash value of fixed length.

[0168] Furthermore, the specific steps of B2 include:

[0169] First, the system receives four inputs: a list of storage nodes, a list of node connections, a complete record of hierarchical positions, and a serialized instruction byte stream. First, it serializes the list of storage nodes and the list of node connections into structured plain text strings, and the complete record of hierarchical positions into a structured Extensible Markup Language (XML) string. Then, it calculates the byte length of each of these three serialized strings and concatenates them in the following fixed order and format: first, write the byte length of the storage node list, then its content; next, write the byte length of the node connection list, then its content; next, write the byte length of the complete record of hierarchical positions, then its content; finally, write the byte length of the instruction serialized byte stream, then write the byte stream itself. A predefined delimiter is used to separate the length and content of each data item, and the concatenation generates a complete scheduling context information block.

[0170] Second, the generated scheduling context information block is used as input and fed into a preset cryptographic hash function for calculation. The cryptographic hash function processes the byte stream of the entire information block and generates a unique binary sequence of fixed length through its internal mathematical transformations. This sequence is the scheduling operation digest hash value, which serves as the unique digital fingerprint of the scheduling credential.

[0171] In this embodiment, the cryptographic hash function adopts the SHA-256 algorithm, and the calculation formula of the SHA-256 algorithm is the prior art in this field and is not an inventive solution of this application, so it will not be described in detail here.

[0172] B3: Obtain the execution completion time of this scheduling operation, generate a scheduling operation timestamp, obtain its hash value from the current end block of the scheduling credential chain as the hash of the preceding block of the system state before this scheduling operation, and combine the scheduling operation digest hash value, scheduling operation timestamp and preceding block hash to form a new block structure to be added to the chain;

[0173] B4: Link the constructed block structure to the end of the existing scheduling credential chain, making it the latest block on the chain, thus completing the writing and updating of the scheduling credential chain.

[0174] The scheduling credential chain introduced in this embodiment uses a cryptographic hash algorithm to perform digest calculations on the updated storage topology, hierarchical location table, and intelligent scheduling instruction set. Combined with timestamps and previous block hashes, it constructs an immutable chained storage structure, ensuring the traceability and data security of every scheduling operation from a technical perspective. This effectively avoids risks such as scheduling behavior tampering and data storage location forgery. More importantly, the historical scheduling performance data accumulated in the scheduling credential chain, such as changes in access latency, cost consumption, and reliability performance, will be continuously fed back to the hierarchical label generation stage. This dynamically optimizes the threshold range, weight allocation, and storage specification parameters of the hierarchical decision model, promoting the continuous iterative upgrade of hierarchical strategies and scheduling rules, and ultimately achieving long-term optimal synergy between storage cost, access performance, and system reliability.

[0175] Example 2:

[0176] Referring to Figure 3, another embodiment of the present invention is provided: a cloud platform system based on intelligent layering, used to implement the data storage and scheduling method based on intelligent layering described in Embodiment 1, comprising:

[0177] The layered label generation module is used to collect multi-dimensional information of the data to be stored on the cloud platform, and generate unique layered labels and layered storage specification tables through feature fusion and layered rule design.

[0178] The data allocation module is used to construct the distributed storage node topology. Based on the precise matching of data storage needs and node resource supply, it completes the reasonable allocation of primary and redundant data replicas and outputs the storage topology structure, hierarchical location table and node running status.

[0179] The instruction generation module is used to model the data scheduling problem as a Markov decision process. By defining the decision space, reward function and solving the optimal strategy, it generates an intelligent scheduling instruction set to achieve dynamic optimization of the storage system.

[0180] The scheduling execution module is used to execute intelligent scheduling instructions to complete data scheduling operations, update the hierarchical location table, write the updated storage topology, hierarchical location table and hash value of intelligent scheduling instruction set into the scheduling credential chain, extract historical effect data from the scheduling credential chain and feed it back to the hierarchical tag generation module to optimize the hierarchical strategy and hierarchical storage specification table.

[0181] The specific functional implementation of each module is described in the relevant content of the intelligent hierarchical data storage and scheduling method in Embodiment 1, and will not be repeated here.

[0182] In some embodiments, the layered label generation module includes:

[0183] The feature extraction unit is used to collect and fuse the core features of the data to form a multidimensional feature vector of the data.

[0184] The hierarchical rule-making unit is used to construct a multi-dimensional decision-making model and formulate label generation rules based on a preset hierarchical strategy.

[0185] The hierarchical label generation unit is used to generate a unique hierarchical label for each data point according to hierarchical rules, establish a data-label association mapping relationship, and output a hierarchical dataset;

[0186] The specification table generation unit is used to formulate a hierarchical storage specification table based on the storage requirements corresponding to the hierarchical tags, and to clarify the storage resource specification parameters corresponding to data of different tags.

[0187] In some embodiments, the data allocation module includes: an attribute acquisition unit, a weight calculation unit, a topology construction unit, and a data matching and allocation unit;

[0188] The attribute acquisition unit is used to collect the physical attributes of all available storage nodes and form a list of node attributes.

[0189] The weight calculation unit is used to calculate the connection weights between storage nodes and generate a node connection weight matrix.

[0190] Topology building units are used to construct tree-like logical topologies and backup-related topologies.

[0191] The data matching and allocation unit is used to accurately match data with storage nodes and allocate replicas.

[0192] The cloud platform system based on intelligent layering provided in this embodiment adopts a modular architecture design and achieves full automation and intelligence of the entire process from data feature extraction, layered label generation, storage node matching, dynamic scheduling execution, to closed-loop optimization through process reengineering. The modular decomposition of the architecture simplifies the system deployment and maintenance difficulty, and the intelligent design of the operation steps, such as automatic topology construction using minimum connection trees, automatic data allocation using weighted matching degrees, and automatic strategy generation using value iteration algorithms, reduces the cost of manual intervention. At the same time, through accurate layered matching to avoid waste of high resources and low demand, dynamic scheduling to reduce ineffective resource consumption, and closed-loop optimization to continuously improve resource utilization, the overall system operation process is not only simpler and more efficient, but also reduces the platform's investment and operating costs from multiple dimensions such as hardware investment, operation and maintenance management, and resource consumption, and has stronger engineering practicality and cost advantages.

[0193] The embodiments of the present invention have been described above with reference to the accompanying drawings. However, the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments under the guidance of the present invention without departing from the spirit and scope of the present invention. All of these variations are within the protection scope of the present invention.

Claims

1. A data storage and scheduling method based on intelligent hierarchical structure, characterized in that, include: S1: Collect data to be stored on the cloud platform, extract multi-dimensional features of the data, formulate label generation rules according to the preset hierarchical strategy, generate a unique hierarchical label with storage heat value for each data, establish a data-label association mapping relationship, and output the hierarchical dataset and the hierarchical storage specification table corresponding to the label. S2: Taking the hierarchical dataset and hierarchical storage specification table as input, the minimum connection tree algorithm is used to construct the distributed storage node topology, allocate data according to the hierarchical specification matching degree, and output the storage topology structure, hierarchical location table and node running status; S3: The data scheduling problem is modeled as a Markov decision process, and the value iteration algorithm is used to intelligently schedule the data to be stored, and output the intelligent scheduling instruction set. S4: Execute the intelligent scheduling instruction set to complete the data scheduling operation, update the hierarchical location table, write the updated storage topology, hierarchical location table and hash value of the intelligent scheduling instruction set into the scheduling credential chain, extract the historical effect data in the scheduling credential chain and feed it back to S1, and optimize the hierarchical strategy and hierarchical storage specification table.

2. The data storage and scheduling method based on intelligent hierarchical structure as described in claim 1, characterized in that, The extraction of multidimensional features from the data specifically includes: obtaining the original file information of the data to be stored, parsing out structured metadata containing file format, file size, creation timestamp, and modification timestamp; based on a preset time window, statistically analyzing the historical access logs of the data to be stored within the time window, calculating the number of accesses, access time distribution characteristics, and the time interval of the most recent access, forming an access feature vector; reading the security policy, backup policy, and compliance policy associated with the data to be stored from the cloud platform's data management policy library, and outputting a security and compliance feature vector containing its security level identifier, backup policy identifier, and compliance requirement description; querying the historical retrieval records of the data to be stored from the cloud platform's user behavior analysis system, calculating its click-through rate, average retrieval response time, and its weight relationship with high-frequency associated search terms, generating a retrieval feature vector; and fusing the structured metadata, access feature vector, security and compliance feature vector, and retrieval feature vector to form a multidimensional data feature vector.

3. The data storage and scheduling method based on intelligent hierarchical structure as described in claim 1, characterized in that, The step of formulating tag generation rules based on a preset hierarchical strategy includes: constructing a hierarchical decision model based on the multi-dimensional feature vector of the data, wherein the decision dimensions include access frequency, security level, disaster recovery level, and retrieval priority; for the access frequency dimension, comparing the number of accesses in the access feature vector with preset cold, warm, and hot access frequency threshold ranges to determine the access heat range to which it belongs, and calculating the access heat value based on the determination result, mapping the access heat value to the corresponding storage heat value tag; for the security level dimension, extracting the security level identifier from the security and compliance feature vector, and determining the encryption algorithm strength, access control list level, and physical or logical separation of storage areas required for data storage based on the security policy mapping table and the security level identifier. For the disaster recovery level dimension, based on the backup strategy identifier in the security and compliance feature vector and according to the preset disaster recovery strategy mapping table, the minimum number of data replicas, the geographical distribution requirements of replica storage, and the time target for data recovery are determined to generate a disaster recovery level tag. For the retrieval priority dimension, based on the click-through rate, response time, and associated search term weights in the retrieval feature vector, a weighted scoring algorithm is used to calculate the retrieval priority score, and a retrieval priority tag is generated according to the interval in which the retrieval priority score is located. The storage heat value tag, security level tag, disaster recovery level tag, and retrieval priority tag are combined according to the preset encoding order to generate the unique hierarchical label, and recorded in the data-tag association mapping relationship.

4. The data storage and scheduling method based on intelligent hierarchical structure as described in claim 1, characterized in that, The process of constructing the distributed storage node topology includes: collecting the physical attributes of all available storage nodes; the physical attributes include node geographical location, total storage capacity and remaining storage capacity, storage media type, network bandwidth, current CPU and memory load rate, and node availability level, forming a node attribute list; based on the node attribute list, calculating the connection weight between any two storage nodes, forming a node connection weight matrix; the calculation of the connection weight comprehensively considers the measured network latency, bandwidth availability ratio, geographical distance, and storage media performance differences, wherein network latency and geographical distance are negatively correlated with the weight, while bandwidth availability and media performance are positively correlated with the weight; using all storage nodes as vertices and the node connection weight matrix as edge weights, applying the minimum connection tree algorithm to construct a tree-shaped logical topology; based on the generated tree-shaped logical topology, according to the preset cross-regional disaster recovery requirements, dynamically associating at least one backup node that meets the minimum geographical distance threshold and network quality threshold from the node attribute list for each primary storage node, establishing backup logical connections, and forming a distributed storage node topology.

5. The data storage and scheduling method based on intelligent hierarchical structure as described in claim 1, characterized in that, The process of allocating data based on hierarchical specification matching degree and outputting storage topology, hierarchical location table, and node operating status includes: for each data block in the hierarchical dataset, encapsulating the storage resource specification parameters in the hierarchical storage specification table corresponding to its unique hierarchical label into a storage demand vector for that data block; simultaneously, based on the node attribute list, extracting the remaining storage capacity, storage media type, network bandwidth, availability level, and geographical location information of each storage node, and encapsulating them into a storage supply vector; calculating a multi-dimensional matching degree score between the storage demand vector of each data block and the storage supply vector of each storage node using weighted Euclidean distance, and selecting the storage node with the highest matching degree score for each data block as the primary storage node for data allocation. The process involves writing data and selecting the node with the second-highest matching score and meeting the disaster recovery geographical distribution requirements from the remaining storage nodes, based on the minimum number of replicas required by the disaster recovery level label in the hierarchical label of the data block. This process completes the replica writing. The allocation results are recorded, and a hierarchical location table is generated. This table records the primary storage node identifier, all redundant replica storage node identifiers, specific storage paths, and allocation timestamps for each data block. Simultaneously, based on the dynamic updates of the node attribute list, a node running status table is generated, which records the current used capacity, CPU and memory load rates, network bandwidth utilization, and online status of each storage node in real time. Finally, the distributed storage node topology, hierarchical location table, and node running status table are output.

6. The data storage and scheduling method based on intelligent hierarchical structure as described in claim 1, characterized in that, The data scheduling problem is modeled as a Markov decision process, comprising: a scheduling state space for the Markov decision process consisting of storage topology, a hierarchical location table, and node operating states. The storage topology describes the logical connections and disaster recovery relationships between nodes; the hierarchical location table records the storage locations of the current primary and redundant copies of each data block; and the node operating states characterize the real-time resource load and availability of each storage node. A set of scheduling actions is defined for the Markov decision process. This set of actions includes at least: migrating a specified data block from its current storage node to another target storage node; creating a new redundant copy of the specified data block on the target storage node; deleting a redundant copy of the specified data block; adjusting the index structure of the specified data block in the retrieval system; and addressing the scheduling action set. For each action in the process, a state transition probability is constructed. The state transition probability function quantifies the probability that the system state will transition from the current state to the next state after executing any specific action, based on a historical statistical model. The probability mainly depends on three dynamic factors: the probability of changes in storage node resource utilization, the probability of changes in network link status, and the probability of changes in data access patterns. A comprehensive instant reward function is defined. The comprehensive instant reward function contains at least four weighted components: a cost-saving reward component inversely proportional to the overall system storage cost; a performance-improvement reward component inversely proportional to the average data access latency; a reliability reward component proportional to the overall system availability and data integrity; and an operation overhead penalty component proportional to the network bandwidth and computing resources consumed by the action.

7. The data storage and scheduling method based on intelligent hierarchical structure as described in claim 1, characterized in that, The intelligent scheduling of the data to be stored using the value iteration algorithm includes: starting from the initially assumed state value, using the defined state transition probability function and a comprehensive immediate reward function, iteratively updating the estimated value function of each state in the entire scheduling state space; when the value function updates of all states tend to stabilize, i.e. convergence, selecting the scheduling action that maximizes its long-term cumulative reward for each state, and outputting a set of optimal scheduling strategies; transforming the optimal scheduling strategies into executable operation instructions according to the current scheduling state space, and finally outputting an intelligent scheduling instruction set; the operation instructions define specific data migration operations, replication operations, or index update operations.

8. The data storage and scheduling method based on intelligent hierarchical structure as described in claim 1, characterized in that, The execution of the intelligent scheduling instruction set to complete the data scheduling operation specifically includes: reading the input intelligent scheduling instruction set, parsing parameters such as the operation type, target data block identifier, and source / target storage node identifier of each instruction, sorting all instructions according to the preset execution priority of the instruction type and the potential data dependencies between instructions, and generating an executable instruction queue; the scheduler starts from the head of the generated executable instruction queue and sequentially retrieves and executes each instruction; when the instruction is for data migration, the scheduler first looks up the current primary storage node identifier of the specified data block in the hierarchical location table, then the scheduler reads the specified data block from that storage node, transmits it to the target storage node via the network, and after the target storage node completes the data writing and verification, the scheduler updates the hierarchical location table. The scheduler modifies the primary storage node identifier of the data block to the target storage node and removes the original location record. When the instruction is to create a replica, the scheduler creates a complete replica of the data block on the target storage node specified in the instruction. After the replica is created and verified successfully, the scheduler updates the hierarchical location table and adds the identifier of the target storage node to the list of redundant replica storage nodes for the data block. When the instruction is to update an index, the scheduler searches for and updates all index entries related to the data block in the distributed retrieval index according to the data block identifier specified in the instruction, ensuring that the storage location and other attributes it points to are consistent with the records in the current hierarchical location table. After all instructions have been executed, the scheduler synchronously updates the system status record according to the instruction execution results, and triggers a sampling update of the node running status.

9. The data storage and scheduling method based on intelligent hierarchical structure as described in claim 1, characterized in that, The step of writing the updated storage topology, hierarchical location table, and hash values ​​of the intelligent scheduling instruction set into the scheduling credential chain specifically includes: extracting the core graph theory representation (i.e., the list of storage nodes and the list of node connections) from the updated storage topology generated after the scheduler completes the data scheduling operation; extracting the complete hierarchical location record containing all data block identifiers, primary storage nodes, redundant replica storage nodes, and storage paths from the updated hierarchical location table; and extracting the instruction serialization byte stream from the intelligent scheduling instruction set on which this execution is based. The extracted storage node list, node connection list, complete hierarchical location record, and instruction serialization byte stream are then processed according to a predefined... The data is concatenated in the specified format to form a scheduling context information block. A cryptographic hash function is applied to the scheduling context information block to generate a unique scheduling operation digest hash value of fixed length. The execution completion time of this scheduling operation is obtained, and a scheduling operation timestamp is generated. The hash value of the current last block of the scheduling credential chain is obtained as the hash of the preceding block of the system state before this scheduling operation. The scheduling operation digest hash value, scheduling operation timestamp, and preceding block hash are combined to form a new block structure to be added to the chain. The constructed block structure is linked to the end of the existing scheduling credential chain to make it the latest block on the chain, completing the writing and updating of the scheduling credential chain.

10. A cloud platform system based on intelligent layering, used to implement the data storage and scheduling method based on intelligent layering as described in any one of claims 1-9, characterized in that, include: The module includes a hierarchical label generation module, a data allocation module, an instruction generation module, and a scheduling and execution module. The layered label generation module is used to collect multi-dimensional information of the data to be stored on the cloud platform, and generate unique layered labels and layered storage specification tables through feature fusion and layered rule design. The data allocation module is used to construct a distributed storage node topology. Based on the precise matching of data storage needs and node resource supply, it completes the reasonable allocation of primary and redundant data replicas and outputs the storage topology structure, hierarchical location table, and node operating status. The instruction generation module is used to model the data scheduling problem as a Markov decision process, use the value iteration algorithm to intelligently schedule the data to be stored, and generate an intelligent scheduling instruction set. The scheduling execution module is used to execute intelligent scheduling instructions to complete data scheduling operations, update the hierarchical location table, write the updated storage topology, hierarchical location table and hash value of intelligent scheduling instruction set into the scheduling credential chain, extract historical effect data from the scheduling credential chain and feed it back to the hierarchical tag generation module to optimize the hierarchical strategy and hierarchical storage specification table.

Images