A data scheduling method and device, a storage medium and an electronic device

Abstract

Embodiments of the present specification disclose a data scheduling method and device, a storage medium and an electronic device, wherein the method comprises: obtaining metadata attribute features and historical access statistical features of a candidate data object in a first storage system, inputting the metadata attribute features and the historical access statistical features into a pre-trained internal table demand prediction model, outputting an acceleration gain prediction value corresponding to the candidate data table, selecting a target object to be accelerated from the candidate data object based on the acceleration gain prediction value, performing an automatic data migration task to synchronize data entities of the target object to be accelerated from the first storage system to a second storage system, and establishing an internal storage table corresponding to the target object to be accelerated in the second storage system. The embodiments of the present specification can improve the data usage efficiency.

Description

Technical Field

[0001] This specification relates to the field of computer technology, and in particular to a data scheduling method, apparatus, storage medium, and electronic device. Background Technology

[0002] In the field of big data processing and analysis, with the rapid development of internet service platform business services, servers generate massive amounts of user interaction logs and system operation data every day. To ensure persistent data storage and full-scale analysis capabilities, the industry typically uses offline computing platforms based on distributed architectures as core data warehouses. However, these offline computing platforms are mainly designed for large-scale batch computing tasks and suffer from I / O bottlenecks when handling ad-hoc queries and high-concurrency data retrieval. Query response latency often reaches minutes or even hours, making it difficult to meet the timeliness requirements of online platform services.

[0003] To improve data query efficiency, related technologies typically introduce high-performance real-time analytics engines based on in-memory computing or columnar storage as an acceleration layer for offline warehouses. Currently, data scheduling in this heterogeneous storage architecture mainly relies on manual configuration by operations personnel based on business experience, or mechanical migration based on simple static rules (such as synchronizing only all data for the day). This management approach lacks the ability to dynamically perceive changes in data access frequency, often resulting in newly generated hot data failing to receive timely query acceleration, while a large amount of historically cold data with extremely low access frequency occupies the expensive storage and computing resources of the real-time engine for extended periods, causing serious resource waste and high system maintenance costs. Summary of the Invention

[0004] This specification provides a data scheduling method, apparatus, storage medium, and electronic device, which can improve data utilization efficiency. The technical solution is as follows:

[0005] In a first aspect, embodiments of this specification provide a data scheduling method applied to a data processing platform including a first storage system and a second storage system, wherein the query response latency of the second storage system is lower than that of the first storage system, and the method includes:

[0006] Obtain the metadata attribute characteristics and historical access statistics of candidate data objects in the first storage system;

[0007] The metadata attribute features and the historical access statistics features are input into a pre-trained internal table demand prediction model, and the acceleration gain prediction value corresponding to the candidate data table is output.

[0008] Based on the predicted acceleration gain value, the target objects to be accelerated are selected from the candidate data objects;

[0009] An automated data migration task is performed to synchronize the data entity of the target object to be accelerated from the first storage system to the second storage system, and an internal storage table corresponding to the target object to be accelerated is established in the second storage system.

[0010] Secondly, embodiments of this specification provide a data scheduling apparatus applied to a data processing platform including a first storage system and a second storage system, wherein the query response latency of the second storage system is lower than that of the first storage system, and the method apparatus includes:

[0011] The acquisition module is used to acquire the metadata attribute characteristics and historical access statistics characteristics of candidate data objects in the first storage system;

[0012] The prediction module is used to input the metadata attribute features and the historical access statistics features into a pre-trained internal table demand prediction model and output the acceleration gain prediction value corresponding to the candidate data table.

[0013] The scheduling module is used to select target objects to be accelerated from the candidate data objects based on the acceleration gain prediction value, execute an automated data migration task to synchronize the data entities of the target objects to be accelerated from the first storage system to the second storage system, and establish an internal storage table corresponding to the target objects to be accelerated in the second storage system.

[0014] Thirdly, embodiments of this specification provide a computer storage medium storing a plurality of instructions adapted for loading by a processor and executing the above-described method steps.

[0015] Fourthly, this specification provides a computer program product storing at least one instruction adapted to be loaded by a processor and to execute the method steps of one or more embodiments of this specification.

[0016] Fifthly, this specification provides a computer program product storing at least one instruction adapted to be loaded by a processor and to execute the method steps of one or more embodiments of this specification.

[0017] Fifthly, embodiments of this specification provide an electronic device that may include: a processor and a memory; wherein the memory stores a computer program adapted to be loaded by the processor and to execute the above-described method steps.

[0018] The beneficial effects of the technical solutions provided in some embodiments of this specification include at least the following:

[0019] In one or more embodiments of this specification, a technique is employed to automatically acquire the metadata attribute features and historical access statistics of candidate data objects in the first storage system, and input these features into a pre-trained internal table demand prediction model to output quantified acceleration gain prediction values. This effectively solves the technical limitations of related technologies where data layer management mainly relies on manual configuration by maintenance personnel based on business experience, leading to strong subjectivity in the decision-making process, delayed response, and inability to adapt to dynamic changes in massive data access. This achieves automated decision-making throughout the entire lifecycle of data, from offline storage to online accelerated storage, reducing system maintenance manpower costs while enabling real-time capture of changes in business hotspots to ensure the timeliness of data analysis. Furthermore, based on the acceleration gain prediction values, an automated data migration task is executed to synchronize the data entities of the target objects to be accelerated to the second storage system to establish internal storage tables. In massive data scenarios, ad-hoc queries are not directly executed on the first storage system, thus avoiding the technical limitations of high disk I / O overhead, low scanning efficiency, long response time, and high data redundancy and network bandwidth consumption caused by full data migration. This improves the query response speed of hot data, achieves dynamic perception of changes in data access popularity, and improves the upper-layer data interaction experience. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments or prior art of this specification, the drawings used in the description of the embodiments or prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a flowchart illustrating a data scheduling method provided in an embodiment of this specification;

[0022] Figure 2 This is a flowchart illustrating an embodiment of the internal table demand forecasting model provided in this specification.

[0023] Figure 3 This is a schematic diagram of a data scheduling process provided in an embodiment of this specification;

[0024] Figure 4 This is a schematic diagram of the training process of an internal demand prediction model provided in the embodiments of this specification;

[0025] Figure 5 This is a schematic diagram of a data migration process provided in an embodiment of this specification;

[0026] Figure 6 This is a schematic diagram of the structure of a data scheduling device provided in the embodiments of this specification;

[0027] Figure 7 This is a schematic diagram of the structure of an electronic device provided in the embodiments of this specification. Detailed Implementation

[0028] The technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this specification, and not all embodiments. Based on the embodiments in this specification, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this specification.

[0029] In the description of this specification, it should be understood that the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance. In the description of this specification, it should be noted that, unless otherwise expressly specified and limited, "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or devices. Those skilled in the art can understand the specific meaning of the above terms in this specification based on the specific circumstances. Furthermore, in the description of this specification, unless otherwise stated, "multiple" means two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, and B alone. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship.

[0030] The present specification will now be described in detail with reference to specific embodiments.

[0031] In one embodiment, such as Figure 1 As shown, a data scheduling method is proposed, which can be implemented using a computer program and run on a data scheduling device based on the von Neumann architecture. This computer program can be integrated into applications or run as a standalone utility application. The data scheduling device can be a data processing platform, including but not limited to: personal computers, tablets, server equipment, computing devices, or other processing devices connected to a wireless modem. Terminal devices can have different names in different networks, such as: user equipment, access terminal, user unit, user station, mobile station, mobile station, remote station, remote terminal, mobile device, user terminal, terminal, wireless communication equipment, user agent or user equipment, cellular phone, cordless phone, equipment in 5G networks or future evolved networks, etc.

[0032] Specifically, the data scheduling method includes:

[0033] S102: Obtain the metadata attribute characteristics and historical access statistics of candidate data objects in the first storage system;

[0034] The first storage system is used for persistent storage and offline batch computing of massive amounts of data. Its characteristics typically include low storage cost and support for data scales exceeding petabytes (PB), but it suffers from high I / O latency when handling high-concurrency ad-hoc queries. In this embodiment, the first storage system can be an offline data warehouse based on a distributed file system (such as an offline MaxCompute system, Hive system, etc.).

[0035] The second storage system is used for online real-time analysis and low-latency queries. Its characteristics typically include in-memory computing or columnar storage optimization, support for high-concurrency read / write operations, and fast query response times (usually milliseconds or seconds), but the cost per unit of storage resource is relatively high. In this embodiment, the second storage system can be an interactive analysis engine or a real-time database (e.g., an online hologres system).

[0036] Candidate data objects can be understood as data entities stored in the first storage system; they can be complete data tables or specific partitions within partitioned tables. In practical applications, candidate data objects are the smallest granular unit for evaluating whether storage tier migration is necessary.

[0037] Metadata attributes refer to data that describes the physical properties of a data table object itself, typically stored in the file system's metadata nodes or a metadata database. These include, but are not limited to: physical storage capacity (e.g., size in bytes), number of file fragments, creation time, last modification time, and the space of the project it belongs to. Metadata attributes reflect the physical cost of data migration.

[0038] Historical access statistics refer to statistical data describing the behavior of data table objects in queries or access by business systems or users over a past period. These include, but are not limited to: read frequency within a specific time window (e.g., the last 7 days, 30 days), average data scan volume involved in read operations, and the timestamp of the last access. Historical access statistics reflect the real-time popularity and access value of the data.

[0039] As an illustration, S102 can be executed by an automated data acquisition module deployed on the control node, aiming to transform unstructured or semi-structured information scattered throughout the storage system into computer-recognizable feature vectors. To adapt to storage systems with different underlying architectures, this step can be performed using either direct extraction based on metadata service interfaces or analysis and aggregation based on operation logs.

[0040] In one feasible implementation, the first storage system (such as a big data warehouse with unified metadata management services) can have a built-in, comprehensive metadata indexing service. The data processing platform establishes a communication connection with the metadata service of the first storage system through a standard database connection protocol (such as JDBC) or the system's open API interface, and traverses all table names under a specified project space to construct a candidate list. For each candidate data table in the list, the platform calls the metadata service's query command to directly read the physical attributes of the data table object, such as its physical size, number of rows, and number of files, as metadata attribute features. Simultaneously, the data processing platform accesses the storage system's built-in audit statistics table or popularity statistics view. These tables are typically updated asynchronously by the storage system's backend and record the access count of data objects within a preset period. The data processing platform directly reads the access count field for the candidate data table over the most recent N days (e.g., 7 days, 30 days) from the aforementioned statistics table as historical access statistics features. This implementation utilizes the storage system's native statistical capabilities, has extremely low system overhead, and can quickly complete feature collection for massive amounts of data tables.

[0041] In one feasible implementation, the first storage system can provide basic file storage or address scenarios lacking readily available statistical metrics. In such cases, access characteristics can be reconstructed through log mining. The data processing platform periodically pulls historical query logs or audit logs from the log server of the first storage system. Then, a log parsing process is initiated, parsing each structured query statement (SQL) in the logs using an Abstract Syntax Tree (AST) to accurately identify the underlying table names actually referenced by the query statement and the partition key values ​​involved. Based on this, a sliding window aggregation calculation is performed on the parsed log data according to a preset time window logic (e.g., the last 7 days, the last 30 days, etc.). During the aggregation process, the cumulative number of times a specific candidate data table is referenced within each window can be counted, and the average of the input fragment size or scanned bytes recorded in the logs can be calculated to generate historical access statistical characteristics. For metadata attribute characteristics, the executing entity calls the operation instructions of the underlying file system (such as the du instruction in a distributed file system) to directly obtain the physical disk space occupied by the data table object.

[0042] S104: Input the metadata attribute features and the historical access statistics features into the pre-trained internal table demand prediction model, and output the acceleration gain prediction value corresponding to the candidate data table.

[0043] The internal table demand prediction model refers to a pre-trained machine learning algorithm model. Through pre-training, this model can be configured to receive metadata attribute features and historical access statistical features (or input data vectors), and output a quantitative result representing the necessity of storing the data object in the second storage system. This internal table demand prediction model establishes a mapping relationship between data features and system performance gains by learning the distribution of historical data.

[0044] The acceleration gain prediction is a quantitative metric output by the internal demand forecasting model, used to measure the expected contribution to overall system performance after migrating a specific data object to a second storage system. This acceleration gain prediction can be a probability value representing the level of demand (such as a floating-point number between 0 and 1) or an absolute value representing the expected savings in computing resources (such as estimated reductions in I / O time).

[0045] In one feasible implementation, a machine learning model, such as logistic regression, can be used as the core of the prediction model. This model receives metadata attribute features and historical access statistics, and performs preprocessing and standardization operations on them. Specifically, methods such as max-min normalization or Z-score standardization are used to map physical storage capacity features (e.g., TB-level) and access frequency features (e.g., times / day) with large numerical ranges to a unified numerical range to obtain the input data feature vector, thus eliminating the interference of dimensional differences on the model weights. The processed input data feature vector is then input into a pre-trained internal table demand prediction model. This model internally stores model weight parameters obtained through training with historical samples, corresponding to the importance of dimensions such as table size, short-term access frequency, and long-term access frequency. A linear weighted summation operation is performed through the internal table demand prediction model to calculate the weighted sum of the features. This weighted sum is then mapped to a value between 0 and 1 using the Sigmoid activation function. This value is the acceleration gain prediction value, which physically represents the confidence probability that the candidate data table is currently in a "hot data" state and is worth migrating to high-performance storage media. For example, the closer the value is to 1, the more the model determines that the table is not only accessed frequently, but also has a moderate physical size, and the migration can significantly improve the overall query hit rate of the system.

[0046] S106: Based on the acceleration gain prediction value, select the target object to be accelerated from the candidate data objects;

[0047] The target object to be accelerated refers to the set of data to which data migration operations will be performed. This set is a subset of the candidate data objects.

[0048] In this embodiment, S106 can be executed by the decision scheduling module of the data processing platform. In some embodiments, considering that the second storage system (such as a real-time interactive analysis engine) typically uses expensive storage media and has limited memory resources, it is advisable not to migrate all data with high predicted values. Based on this, a fast filtering strategy based on a fixed threshold or a greedy filtering strategy based on dynamic resource constraints can be adopted according to different business scenarios and resource conditions.

[0049] In one specific implementation, a fixed threshold judgment logic can be used. The data processing platform reads the preset admission threshold (e.g., 0.8) from the system configuration file. The data processing platform traverses all candidate data objects and their corresponding acceleration gain prediction values. For each object, the acceleration gain prediction value is numerically compared with the admission threshold. If the prediction value is greater than or equal to the admission threshold, the data object is determined to have significant acceleration value, and its unique identifier is added to the list of target objects to be accelerated; if the prediction value is less than the admission threshold, it is marked as cold data to be retained in the first storage system.

[0050] In one specific implementation, a capacity-aware dynamic sorting and filtering logic is employed. The data processing platform queries the current disk utilization and remaining available storage capacity of the second storage system via an API interface, and sets a safety waterline (e.g., 85% of the total capacity) based on the system operation and maintenance strategy. The data processing platform obtains the physical size information of all candidate data tables and sorts all candidate data tables in descending order according to the acceleration gain prediction value output by S104, generating a priority queue. The data processing platform initializes an accumulated capacity counter and sequentially traverses the data tables in the priority queue. During the traversal, it simulates the total space occupied after the current data table is moved in, i.e., it determines whether the sum of the current accumulated capacity and the current table's physical size exceeds the set safety waterline. If it does not exceed the safety waterline, the table is added to the target objects to be accelerated, and the accumulated capacity counter is updated; if it exceeds the safety waterline, it means that the remaining space is insufficient to accommodate the high-priority table, and the platform can choose to skip the table and try to accommodate a smaller subsequent table, or directly terminate the filtering process. This implementation essentially takes into account scenarios where the second storage system has limited storage resources or requires strict cost control. It solves the problem of maximizing value limited by the storage backpack capacity, ensuring that, under the premise of limited physical resources, the data objects with the highest unit benefit are accelerated first, thereby achieving the optimal balance between overall system performance and hardware cost.

[0051] S108: Execute an automated data migration task to synchronize the data entity of the target object to be accelerated from the first storage system to the second storage system, and establish an internal storage table corresponding to the target object to be accelerated in the second storage system.

[0052] Data entity: refers to a collection of data blocks or files stored on a physical medium. In the first storage system, it can be represented as file blocks distributed across multiple data nodes; in the second storage system, it can be represented as storage units after columnar compression and encoding.

[0053] Automated data migration tasks refer to a series of scripts or instruction sets that are automatically generated and executed by the system scheduling engine, typically including but not limited to data definition, data extraction, data transformation, and data loading.

[0054] In one specific implementation, a full table migration can be performed. The data processing platform reads the schema information (including field names, field types, partition keys, etc.) of the target object to be accelerated from the first storage system through the heterogeneous data source connector. Based on the syntax specifications of the second storage system, the corresponding table creation statements are automatically generated. When executing the table creation statements, the table structure parameters are automatically optimized according to the feature information output by S104: for example, bitmap indexes are created for the fields with the highest query frequency, dictionary encoding is set for high-cardinality numeric segments, and the data table is configured as a row-based or column-based storage format to accommodate point-to-point or aggregate-based query requirements. After table creation is completed, the data processing platform generates a data synchronization script to establish a high-concurrency data transmission channel, reading all data entities from the first storage system in parallel and writing them to the internal storage table of the second storage system. After the task is completed, the data processing platform automatically verifies the row count and checksum of the data on both sides to ensure data consistency.

[0055] In one specific implementation, a partition-level fine-grained hybrid migration model can be adopted. The data processing platform does not migrate the entire table; instead, it only creates corresponding physical partition sub-tables in the second storage system for data entities identified as hot partitions (e.g., data partitions from the last 30 days). For the remaining cold partition data, the data processing platform only creates an external mapping table in the second storage system pointing to the first storage system. Subsequently, the data processing platform creates a unified logical view or parent table in the second storage system. Utilizing the database's partitioning and pruning capabilities, query requests for recently hot data are routed to the local high-speed internal storage table, while query requests for historically cold data are routed via the network to the remote external mapping table. This approach constructs a hybrid storage architecture combining internal and external tables, achieving a balance between storage cost and query performance without the user's awareness.

[0056] In one or more embodiments of this specification, a technique is employed to automatically acquire the metadata attribute features and historical access statistics of candidate data objects in the first storage system, and input these features into a pre-trained internal table demand prediction model to output quantified acceleration gain prediction values. This effectively solves the technical limitations of related technologies where data layer management mainly relies on manual configuration by maintenance personnel based on business experience, leading to strong subjectivity in the decision-making process, delayed response, and inability to adapt to dynamic changes in massive data access. This achieves automated decision-making throughout the entire lifecycle of data, from offline storage to online accelerated storage, reducing system maintenance manpower costs while enabling real-time capture of changes in business hotspots to ensure the timeliness of data analysis. Furthermore, based on the acceleration gain prediction values, an automated data migration task is executed to synchronize the data entities of the target objects to be accelerated to the second storage system to establish internal storage tables. In massive data scenarios, ad-hoc queries are not directly executed on the first storage system, thus avoiding the technical limitations of high disk I / O overhead, low scanning efficiency, long response time, and high data redundancy and network bandwidth consumption caused by full data migration. This improves the query response speed of hot data, achieves dynamic perception of changes in data access popularity, and improves the upper-layer data interaction experience.

[0057] Please see Figure 2 , Figure 2 This is a flowchart illustrating a method for using an internal table demand prediction model as proposed in this specification. Specifically, the process of inputting the metadata attribute features and the historical access statistics features into a pre-trained internal table demand prediction model and outputting the acceleration gain prediction value corresponding to the candidate data table can be performed in the following way:

[0058] S202: Extract the physical storage capacity features of the candidate data objects based on the metadata attribute features, and calculate the data access frequency features of the candidate data objects in multiple time windows based on the historical access statistics features;

[0059] Physical storage capacity characteristics refer to the features that characterize the amount of space occupied by candidate data objects in the underlying physical medium of the first storage system. Examples include table size and partition size.

[0060] Multiple time windows refer to pre-defined continuous statistical intervals of different lengths over a time dimension. For example, a short-term window corresponds to the most recent 7 days, reflecting immediate surges in popularity; a long-term window corresponds to the most recent 90 days, reflecting historical access habits.

[0061] In the first specific implementation, the data processing platform connects to the metadata database of the first storage system, directly reads physical storage capacity characteristic fields such as the total size and original data size stored in the records, extracts their values, and converts them into a unified unit as the physical storage capacity characteristic. Simultaneously, regarding the data access frequency characteristic, the audit log table is queried using a preset fixed time window logic, and then data statistical aggregation is performed. The number of data operation records for the candidate data object is counted under multiple time windows (such as the last 7 days, the last 30 days, and the last 90 days), and the data access frequency characteristic of the candidate data object under multiple time windows is determined based on this number of data operation records.

[0062] S204: Generate a basic dimension feature vector based on the combination of the physical storage capacity feature and the data access frequency feature;

[0063] The basic dimension feature vector refers to the numerical sequence formed by splicing together heterogeneous physical attribute data and access statistics data after feature engineering transformation (such as normalization and discretization), which can be directly processed by machine learning models.

[0064] In one feasible implementation, considering that data table sizes typically exhibit a long-tail distribution (from KB to PB), the data processing platform performs a logarithmic transformation on the physical storage capacity characteristics, i.e., logarithmically smoothing the physical storage capacity characteristics (e.g., calculating log10(size_bytes) to compress the numerical range). For the data access frequency characteristics, a maximum and minimum normalization algorithm is used, utilizing historical maximum and minimum frequencies to map the current frequency to the [0,1] interval. After the numerical transformation, the processed values ​​are sequentially concatenated according to a preset index order (e.g., index 0 represents capacity, and indices 1-3 represent the frequency of different windows) to construct the basic dimensional feature vector.

[0065] Optionally, the multiple time windows include short-term time windows, medium-term time windows, and long-term time windows. Specifically, the generation of the basic dimension feature vector based on the combination of the physical storage capacity characteristics and the data access frequency characteristics can be performed in the following manner:

[0066] Step A2: Normalize the physical storage capacity feature and the data access frequency feature to construct a time-series heat feature group including short-term burst degree, medium-term stability degree and long-term decay degree, and determine the access density feature per unit storage cost based on the physical storage capacity feature.

[0067] Normalization refers to using specific mathematical algorithms, such as minimum and maximum scaling or Z-score standardization, to map raw data with different physical dimensions (such as storage capacity in bytes versus access counts in integers) to a uniform dimensionless numerical range, in order to eliminate the interference of differences in numerical magnitude.

[0068] The time-series popularity feature group refers to a set of dynamic indicators derived from historical access data across multiple time windows, used to characterize the rate and pattern of data access behavior changes over time.

[0069] Short-term burst rate characterizes the surge in access frequency within a recent shorter time window (e.g., 7 days) relative to a medium-term window (e.g., 30 days).

[0070] Medium-term stability characterizes the variance or standard deviation of the frequency of access within a medium-term time window, reflecting the regularity of data access.

[0071] Long-term decline rate characterizes the downward trend of access frequency within a long historical window (such as 90 days), and is used to identify data with a gradual decline in query popularity.

[0072] The access density characteristic of unit storage cost refers to a composite indicator obtained by correlating the access frequency of data with its physical storage consumption. It can be the query I / O throughput generated per unit of storage space occupied.

[0073] To illustrate, the original data is standardized based on its multidimensional characteristics. The physical storage capacity (usually in bytes) of the candidate data object and the access frequency (usually in integers) in each time window are read. Considering the large range between the two values, a strategy combining logarithmic smoothing and max-min normalization is adopted to map the physical storage capacity to the floating-point range of 0 to 1, and the access frequency is scaled on the same scale to eliminate the impact of physical unit differences on subsequent calculations.

[0074] After basic normalization, a time-series popularity feature group is constructed based on a sliding time window mechanism to capture the dynamic trend of data changes. Then, the short-term surge rate is calculated by comparing the ratio or growth slope of the average daily access frequency in the most recent short period (e.g., 7 days) with the average daily access frequency in the medium period (e.g., 30 days) to quantify whether there is a recent surge in popularity. If the ratio is greater than 1, it indicates that the data is in a surge period; otherwise, it is in a stable period. Next, the medium-term stability rate is calculated by statistically analyzing the standard deviation or variance of the daily access frequency in a preset medium period to quantify the dispersion of data access. If the variance is extremely small and less than a preset threshold, it indicates that the data belongs to a normalized query task. Finally, the long-term decline rate is calculated by fitting linear regression or calculating the frequency decay ratio of the most recent 30 days to the past 90 days. If the frequency decay ratio is less than the ratio threshold, it indicates that the popularity of the table is rapidly declining, thereby identifying cold data with high historical access volume but recent stability.

[0075] Meanwhile, to assess the economics of data migration, access density characteristics per unit storage cost are determined based on the physical storage capacity characteristics. This involves calculating the quotient of the cumulative access frequency over a short period and the physical storage capacity of the data object, or calculating the difference between frequency and capacity in the logarithmic domain. This generates a density index characterizing the number of queries contributed per unit of storage space. This index effectively distinguishes between high-frequency small table types (high density, high value) and low-frequency large table types (low density, low cost-effectiveness), providing crucial quantitative evidence for subsequent models to make optimal acceleration decisions under limited storage resource constraints.

[0076] Step A4: Determine the historical query logs corresponding to the historical access statistical features, determine the computational operator type and data scan volume corresponding to the structured query statements in the historical query logs, determine the query complexity score based on the computational operator type and the data scan volume, and use the query complexity score as a computational cost dimension feature.

[0077] Historical query logs are text or binary files stored on non-volatile storage media that record all instructions received and executed by the database system. Their content typically includes execution timestamps, execution time, initiating user, the specific structured query statement, and statistics on the physical resources consumed in executing that statement.

[0078] The computational operator type refers to the specific logical operation category represented by each node in the execution plan tree generated by the database query optimizer. Common computational operator types include, but are not limited to: TableScan (full table scan, I / O intensive), HashJoin (hash join, memory intensive), Sort / Aggregate (sorting / aggregation, CPU intensive), and Filter (filtering, computationally lightweight).

[0079] The query complexity score is a quantitative value calculated based on the product of the computational overhead weight of the operator and the size of the data involved. This value is used to characterize the CPU computing power and memory bandwidth usage of a computer system when executing a specific query statement.

[0080] In one feasible implementation, historical query log fragments related to candidate data objects are extracted from the audit center or log server of the first storage system. Using regular expressions or specific log ID indexing techniques, all structured query statements (SQL) referencing the candidate data table are filtered out. Then, a syntax parser is started to construct an abstract syntax tree for each extracted SQL statement, directly calling the database engine's interface to obtain the physical execution plan of the SQL statement. During this process, the nodes of the syntax tree or execution plan are traversed to identify the key computational operator types involved in the query, and basic weights are assigned to different operators according to a preset resource consumption model. For example, the system marks operators involving multi-table joins or full sorting as high-weight nodes, while marking simple point-to-point lookup operators as low-weight nodes. Simultaneously, the actual data scan volume of the statement is extracted from the log's execution statistics field, and weighted calculation logic is performed to generate a query complexity score.

[0081] Specifically, the system multiplies or linearly weights the weights of each identified operator with the logarithmic form of the data scan volume to obtain the complexity score for a single query. To form stable feature inputs, the system aggregates all query complexity scores of the candidate data table within the statistical period (e.g., by calculating the average or 95th percentile), and standardizes the aggregated result into a computational cost dimension feature. A higher value for this feature usually means more computational resources can be saved by migrating the table to a high-performance secondary storage system, thus achieving a higher speedup gain score in model prediction.

[0082] Step A6: Perform feature engineering based on the time-series heat feature group, the access density feature, the computation cost dimension feature, the physical storage capacity feature, and the data access frequency feature to obtain the basic dimension feature vector.

[0083] Indicatively, the multi-source heterogeneous indicators calculated in the preceding steps are integrated into a standardized mathematical vector to facilitate subsequent reading and inference by the prediction model. Since the input features encompass physical attributes (capacity), statistical attributes (frequency), temporal attributes (explosiveness / decline), and computational attributes (complexity score), these input features differ significantly in numerical magnitude and data distribution. Based on this, the processing first performs a vector concatenation operation on the multi-dimensional features. Following the predefined feature dictionary order, the normalized physical storage capacity feature, data access frequency feature, access density feature, computational cost dimension feature, and each sub-feature from the temporal popularity feature group are sequentially filled into a fixed-length floating-point array to construct the original high-dimensional feature vector.

[0084] After the initial concatenation, to improve the model's training efficiency and prevent collinearity between features from distorting model weights (e.g., access frequency and access density may be highly positively correlated), further spatial transformation and dimensionality reduction are performed on the original feature vectors. Specifically, principal component analysis or linear discriminant analysis algorithms can be used to calculate the eigenvalues ​​and eigenvectors of the original feature covariance matrix, projecting the original high-dimensional feature space into a low-dimensional orthogonal subspace, thereby generating a set of linearly independent new features that retain most of the information variance. Alternatively, L2 regularization can be performed on the concatenated vectors, i.e., calculating the Euclidean norm of the vectors and dividing each element of the vector by the norm, thus constraining all feature vectors to a unit hypersphere. The numerical sequence after the above mathematical transformation and cleaning is confirmed as the basic dimension feature vector. At this point, not only are the dimensional differences eliminated, but the ability of the data to express high-value hotspots is also enhanced through feature fusion, providing high-quality input data for the accurate prediction of subsequent models.

[0085] This specification constructs a multi-dimensional feature vector encompassing time-series popularity trends (boom / decline), access density per unit storage cost, and query computational complexity scores. This effectively overcomes the shortcomings of relying solely on a single static access frequency for decision-making, which leads to the inability to distinguish pseudo-hotspots, identify hidden performance bottlenecks with low frequency but high computational costs, and assess storage cost-effectiveness. This enables a comprehensive and refined evaluation of the value of data acceleration, ensuring that limited secondary storage system resources can accurately target and prioritize key data objects that are in the period of business boom, have the highest storage output ratio, and can significantly save overall system computing power. This maximizes the hardware resource utilization and end-to-end query performance of the tiered storage architecture.

[0086] S206: Input the basic dimension feature vector into the pre-trained internal table demand prediction model, and output the acceleration gain prediction value for the candidate data object.

[0087] In a schematic example, the processed base dimension feature vector is input into a pre-trained internal table demand prediction model. This model internally stores model weight parameters obtained through training on historical samples, corresponding to the importance of dimensions such as table size, short-term access frequency, and long-term access frequency. A linear weighted summation operation is performed on the internal table demand prediction model to calculate the weighted sum of the features. This weighted sum is then mapped to a value between 0 and 1 using the Sigmoid activation function. This value is the predicted acceleration gain.

[0088] For example, the model training process of the internal demand forecasting model is explained below:

[0089] Model creation: Create an initial internal demand forecasting model based on a machine learning model for the internal demand forecasting scenario;

[0090] Sample data acquisition: Acquire a large amount of sample data, which serves as the basic dimension feature vector of the sample.

[0091] Sample data labeling: Based on the internal table demand prediction scenario, the expert service introduces manual labeling of sample data, which includes the acceleration gain prediction value label for each sample data.

[0092] Model training process: Input sample data into the initial internal table demand prediction model for at least one round of model training to obtain the predicted acceleration gain value. Based on the predicted acceleration gain value and the acceleration gain value label, determine the model loss value using a preset model loss function. Based on the model loss value, adjust the model parameters of the initial internal table demand prediction model until the model training termination condition is met to obtain the internal table demand prediction model.

[0093] Optionally, the model's training termination conditions may include, for example, the loss function value being less than or equal to a preset loss function threshold, or the number of iterations reaching a preset threshold. Specific training termination conditions can be determined based on actual circumstances and are not specifically limited here.

[0094] It should be noted that the machine learning models involved in one or more embodiments of this specification include, but are not limited to, fitting of one or more of the following machine learning models: Convolutional Neural Network (CNN) model, Deep Neural Network (DNN) model, Recurrent Neural Networks (RNN) model, Embedding model, Gradient Boosting Decision Tree (GBDT) model, Logistic Regression (LR) model, etc.

[0095] Optionally, referring to steps A2-A6, further, specifically executing the step of inputting the basic dimension feature vector into the pre-trained internal table demand prediction model and outputting the acceleration gain prediction value for the candidate data object, can be done in the following way:

[0096] Step B2: Input the basic dimension feature vector into the pre-trained internal table demand prediction model;

[0097] Step B4: Using the internal table demand prediction model, based on the computational cost dimension feature and data access frequency feature in the basic dimension feature vector, infer the cumulative query saving computational time after migrating the candidate data object to the second storage system, determine the future trend correction coefficient for the cumulative query saving computational time based on the time series heat feature group, and generate a resource occupation penalty value representing the migration cost based on the physical storage capacity feature and the access density feature.

[0098] Cumulative query time savings refer to the total reduction in CPU execution time or disk I / O wait time when processing the same query load, estimated over a future unit of time, if candidate data objects are stored in the second storage system (high-performance tier) instead of the first storage system (low-speed tier). In other words, it represents the positive physical benefit of data migration.

[0099] The future trend correction coefficient is a dimensionless floating-point number (usually greater than 0) derived from time-series heat characteristics. It is used to amplify (predict heat) or reduce (predict cooling) current returns. It reflects the time decay or enhancement effect of returns.

[0100] The resource usage penalty value is a quantitative representation of the cost required to migrate data from the first storage system to the second storage system. This cost includes the network bandwidth resources used during the migration process and the expensive storage media resources (such as SSDs / memory) that are occupied long-term after the migration.

[0101] Positive resource returns refer to the expected total returns after trend correction, which is the time saved if future developments follow the predicted trend.

[0102] Indicatively, the basic dimension feature vector is decomposed into three parallel computational branches—revenue, trend, and cost—through the internal demand forecasting model.

[0103] In the benefit calculation branch, the internal table demand prediction model calls the pre-trained speedup regression parameters and performs a weighted product operation on the computational cost dimension feature (representing the complexity of a single query) and the data access frequency feature (representing the total number of queries) in the basic dimension feature vector. Specifically, the computational cost feature is used to estimate the response time difference of a single query between heterogeneous storage systems (i.e., the time saved per query), and this is multiplied by the normalized access frequency to infer the expected cumulative query computation time saved per unit time after migrating the candidate data object to the second storage system. This value directly reflects the positive physical benefit brought by the migration (i.e., how much computing power is released).

[0104] In the trend correction branch, the control internal demand forecasting model focuses on the time-series heat feature group in the input vector, and processes the short-term explosiveness and long-term decline features through a non-linear activation function (such as Exponential or Softplus function). If the short-term explosiveness value is significantly high, an exponential mapping is used to generate a floating-point number greater than 1.0 as the future trend correction coefficient to amplify the cumulative query calculated above and save computation time, thereby reflecting the prediction that the hot spot is about to explode; conversely, if the long-term decline is high, a coefficient less than 1.0 is generated to suppress the revenue prediction.

[0105] In the cost penalty branch, the internal demand forecasting model performs a cost function calculation based on physical constraints, while the value forecasting model reads the physical storage capacity characteristics as the basic penalty term and adjusts it in conjunction with access density characteristics. During model training, a unit pricing weight for storage resources is set within the model's internal logic. The physical capacity is multiplied by this unit pricing weight to obtain the basic migration cost. If the access density characteristic is extremely low (indicating inflated and hollow data), this cost value is further amplified through a non-linear function to generate the final resource occupancy penalty value. This penalty value will be used as a negative term to offset benefits in subsequent steps, thereby ensuring that the final prediction result meets the optimization objective of minimum resource occupancy and maximum performance improvement.

[0106] Step B6: Using the future trend correction coefficient, the cumulative query time saving calculation is corrected by the internal table demand prediction model to obtain positive resource benefits. Based on the positive resource benefits and the resource occupation penalty, the acceleration gain prediction value is obtained, and the acceleration gain prediction value for the candidate data object is output.

[0107] Positive resource returns refer to the total future returns expected by the system after trend correction. Its physical meaning can be interpreted as "the total effective CPU computing power or I / O time slices that are expected to be released for the entire computer cluster through data migration within a future cycle."

[0108] Resource usage penalty value refers to the quantified cost of storing data in a second storage system.

[0109] Indicatively, step B6 is executed by the output layer or final logic calculation unit of the internal demand forecasting model, essentially completing the final mathematical game of revenue versus cost. The internal demand forecasting model first performs a trend calibration of revenue, then performs a multiplication operation based on the time saved by cumulative queries and the future trend correction coefficient to obtain the product result. If the correction coefficient is greater than 1 (i.e., predicting a surge in popularity), the static revenue is amplified; if the correction coefficient is less than 1 (i.e., predicting a decline in popularity), the static revenue is compressed. This product result is defined as positive resource revenue, representing the system's true expectation of the future value of this data object.

[0110] Simultaneously, the core net benefit calculation is performed, using the positive resource benefit obtained from the above calculation as the minuend and the generated resource occupancy penalty value as the subtrahend, performing arithmetic subtraction or weighted difference calculation. This simulates the net profit process in the physical world, meaning the result is positive only when the predicted improvement in computing performance significantly exceeds the cost of its occupied storage resources. To adapt to different system decision-making mechanisms, after calculating the difference, the model can directly output the original difference as an absolute score in the form of a predicted acceleration gain value for accurate ranking of candidate objects; alternatively, the difference can be mapped to the (0,1) interval using a Sigmoid activation function to output a probabilistic predicted value. Finally, this value is passed to the scheduling control part as the sole quantitative basis for whether to trigger a data migration task, thereby ensuring that every data migration in the system pursues the optimal solution for global resource allocation, rather than blindly accelerating in a localized manner.

[0111] In the embodiments of this specification, a technique is adopted to extract and combine physical storage capacity and access frequency under multiple time windows to generate normalized basic dimension feature vectors, which are then input into a pre-trained internal table demand prediction model for inference calculation. This technique transforms discrete, multi-source underlying physical statistical information into high-precision quantized accelerated gain prediction values, providing scientific, objective, and statistically based data support for subsequent data hierarchical storage decisions, and significantly improving the accuracy of hot data identification and the system's resource scheduling efficiency.

[0112] In one feasible implementation, please refer to Figure 3 , Figure 3This is a flowchart illustrating a data scheduling process. The data scheduling scheme further includes the following steps:

[0113] S302: Periodically synchronize the reference metadata attribute characteristics and reference historical access statistics characteristics of existing reference internal storage tables from the system tables of the second storage system;

[0114] The reference internal storage table refers to a data entity that has completed data migration at the current moment, physically exists in the second storage system (high-performance computing layer), and is currently occupying the valuable storage media and computing resources (such as memory indexes) of that system.

[0115] The reference metadata attribute characteristics refer to the real-time physical status data of the table in the second storage system, which is different from the static metadata of the first storage system. It can reflect the actual storage usage, fragmentation distribution and row size after columnar compression and encoding optimization.

[0116] The historical access statistics feature refers to the statistics based on the operation logs of the second storage system, which reflect the frequency of the table being called by actual business queries in the recent period (such as during the period after acceleration), the timestamp of the most recent access, and the coverage of the query scan.

[0117] This illustration demonstrates the execution of a status monitoring daemon on the data processing platform, aiming to establish a comprehensive understanding of the online acceleration layer's data and provide real-time data support for subsequent lifecycle management. Since the secondary storage system is typically a real-time data warehouse supporting high-concurrency queries (such as Hologres, ClickHouse, or Elasticsearch), it internally maintains a standard system metadata view and active audit logs. The processing begins with a periodic synchronization task triggered by the scheduling engine (e.g., scheduled to execute at 2 AM daily). The executing entity establishes a connection channel with the secondary storage system via JDBC or a high-performance proprietary protocol.

[0118] Subsequently, the data processing platform executes specific SQL query commands or API calls to scan the core system tables of the second storage system. Regarding the reference metadata attribute characteristics: the data processing platform reads the physical file size, primary key index usage, and memory resident size of each reference internal storage table; these metrics directly reflect the table's consumption cost of high-performance hardware resources. Regarding the reference historical access statistics characteristics: the query log table or access statistics view is scanned to aggregate statistics on the total number of SELECT operations, the number of data rows scanned, and the time of the last query within the current statistical period. The data processing platform cleans and structures these two types of characteristic data and updates them to the metadata repository of the management platform, forming the latest health status profile of the reference internal storage table.

[0119] S304: Input the reference metadata attribute features and reference historical access statistics features into the internal table demand prediction model to perform a survival evaluation of the reference internal storage table and calculate the survival necessity probability value.

[0120] The persistence evaluation process refers to a secondary assessment of whether data still has acceleration value after it has already existed in the secondary storage system, based on its actual resource consumption and latest access performance.

[0121] In this embodiment, this step is performed by the value assessment engine deployed on the control node. Its essence is to use the latest feedback data (referencing metadata attribute characteristics and historical access statistics characteristics) to conduct a second audit on the rationality of the configuration of existing resources. For the internal table demand prediction model, its survival evaluation process is also the execution of the forward propagation process, but the output result is called the survival necessity probability value at this time.

[0122] Furthermore, the system maps the reference metadata attribute features and reference historical access statistics features obtained synchronously in step S302 into a vector format compatible with the model input layer to obtain the basic dimension feature vector. The pre-trained internal table demand prediction model is then invoked to perform forward propagation inference calculations based on the basic dimension feature vector. During the forward propagation inference process, the model outputs the probability value of continued existence necessity.

[0123] S306: If the probability value of continued existence indicates that acceleration is not required, then an automated cleanup task is performed to delete the corresponding reference internal storage table from the second storage system.

[0124] "No acceleration required" refers to the data classification status determined by the output of the internal table demand prediction model. It usually corresponds to a situation where the probability value of continued necessity is lower than a preset eviction threshold (such as 0.3 or 0.2), indicating that the current access benefits of the data can no longer cover its holding costs in the second storage system.

[0125] An automated cleanup task refers to a sequence of instructions automatically generated by the system scheduling engine to perform physical deletion operations and reclaim resources in the second storage system. This task only affects the replica data in the acceleration layer and does not affect the original data in the first storage system.

[0126] Indicatively, S306 can be automatically triggered by the operation and maintenance execution module of the data processing platform, aiming to achieve data updates in a tiered storage architecture, i.e., timely release of valuable high-performance resources occupied by cold data. The probability value of the step's necessity is compared with a preset lifecycle eviction threshold. If the probability value is lower than the threshold, the reference internal storage table is considered to have lost its economic viability in residing in the real-time layer and is marked as not requiring acceleration. Otherwise, no processing is performed.

[0127] Furthermore, the data processing platform generates and executes an automated cleanup task. To ensure data security and business continuity, this task may include two sub-steps: security verification and physical deletion. Security verification may involve accessing the metadata service of the first storage system to verify the integrity and availability of the original copy of the data object in an offline, low-cost storage tier (such as HDFS or ODPS), ensuring that the deletion operation only removes the accelerated view and does not result in permanent data loss. After successful verification, the system sends standard DDL commands to the second storage system through the database management interface.

[0128] Furthermore, the data processing platform performs a physical deletion operation on the second storage system, unloading the table's index structure from memory and erasing the corresponding data file blocks from high-speed storage media such as SSDs. With the cleanup task complete, the system synchronously updates the metadata status on the management side, changing the storage status of the data object from online acceleration to offline archiving, and adding the released physical storage capacity back into the available resource pool to provide sufficient storage space for subsequent new hot data predicted to be of high value (such as tables from upcoming major promotional events).

[0129] This specification establishes a closed-loop data lifecycle management mechanism based on real load feedback. It utilizes models to periodically and dynamically reassess the necessity of online acceleration data (technical means), effectively solving the technical problems of storage resources being occupied by a large amount of outdated cold data and the high risk and lag of manual cleanup (technical issues). This enables automated updates and fragmentation recycling of high-performance storage space, ensuring that limited hardware resources can be continuously recycled to support new business hotspots, and maintaining the long-term healthy operation and extremely high resource utilization of the tiered storage architecture.

[0130] In one feasible implementation, please refer to Figure 4 , Figure 4 This is a flowchart illustrating the training process of an internal demand forecasting model, including:

[0131] S402: Select the data table that has been configured as an internal storage table in the second storage system as a positive sample, and randomly select a preset proportion of data tables from the data tables that have not been configured as internal storage tables in the first storage system as negative samples;

[0132] The purpose of implementing S402 is to construct a supervised learning dataset from a real production environment that is both physically representative and conforms to statistical distribution patterns. In actual big data storage architectures, the amount of long-term accumulated cold data often far exceeds the amount of active hot data (e.g., 100,000 cold tables versus 500 hot tables). If the entire dataset is used directly for training, the model will be severely biased towards prediction and will not be accelerated.

[0133] First, positive sample anchoring is performed. The system directory of the second storage system (such as Hologres or ClickHouse) is scanned through the metadata interface to identify all internal tables that are currently active and have a physical storage structure. The metadata and historical access characteristics of these tables are then extracted and uniformly labeled as positive samples (e.g., Label=1). This operation ensures that the model can learn the common characteristics of data that has been validated as high-value.

[0134] Then, negative sample filtering and extraction are performed. The system scans the full metadata list of the primary storage system (such as MaxCompute or HDFS). First, through set difference operations, data tables that already appear in the positive sample list are eliminated, thus defining a candidate pool of pure cold data. Based on a preset positive-to-negative sample ratio (e.g., 1:4), the target number of negative samples to be extracted is calculated. A specified number of data tables are randomly selected from the candidate pool, and the feature data of these selected tables are extracted and labeled as negative samples (Label=0). In this way, a training dataset with a relatively balanced class distribution is constructed, which preserves the diversity of cold data and avoids model failure caused by data skew.

[0135] S404: Divide the positive and negative samples into a training group and a validation group, and use the training group to train the initial internal table demand prediction model for at least one round to establish a linear mapping relationship between the input data and the internal table demand label, and use the model activation function to map the output of the linear mapping relationship into a probability value, and define the probability value as the acceleration gain prediction value.

[0136] A linear mapping relationship refers to the direct proportional relationship between input features and output scores established internally by matrix multiplication within the model. Physically, this is represented by the dot product of feature vectors and weight vectors.

[0137] The model activation function refers to a non-linear mathematical transformation (such as Sigmoid, Softmax) used to compress unbounded linear scores into a fixed interval (0,1) to give them probabilistic statistical significance.

[0138] Indicatively, this step is executed by the model training engine deployed on a high-performance computing cluster. Its essence is to search and optimize the internal parameter space of the mathematical model using large-scale sample data. The process begins with dataset splitting. The system employs a stratified random sampling algorithm to divide the positive and negative sample sets constructed in step S402 into training and validation groups according to a preset ratio, ensuring consistency in the positive and negative sample distribution between the two groups.

[0139] The iterative training phase begins by initializing an initial internal table demand prediction model based on a machine learning model, containing an undetermined weight parameter matrix (W) and a bias term (b). In each iteration, feature vector data from the training group is read in batches and input into the initial internal table demand prediction model. The model performs forward propagation to establish a linear mapping: the input feature vectors are multiplied by the current weight parameter matrix, and the bias term is added to obtain a linear score (Logits) in a continuous numerical domain. To transform this abstract linear score into a decision criterion with clear physical meaning, the system calls the model activation function (e.g., a preset Sigmoid function) to perform a non-linear compression mapping on the linear score, converting it into a floating-point number between 0.0 and 1.0. This floating-point number mathematically represents the confidence level (usually a predicted probability value) that the sample belongs to "high-value hot data." In some scenarios, it can be defined as an acceleration gain prediction value, and in others, as a survival necessity probability value.

[0140] During the backpropagation training process, the system further calculates the model training loss based on the predicted probability value and the true label using a preset loss function (such as hinge loss, cross-entropy loss, Euclidean distance loss, etc.), and fine-tunes the weight parameter matrix along the gradient descent direction through the backpropagation algorithm, thereby gradually strengthening the logical relationship between the input features and the internal table demand label, until the internal table demand prediction model can accurately capture the inherent law between data features and acceleration value and meet the model end training conditions.

[0141] Optionally, the model's training termination conditions may include, for example, the loss function value being less than or equal to a preset loss function threshold, or the number of iterations reaching a preset threshold. Specific training termination conditions can be determined based on actual circumstances and are not specifically limited here.

[0142] It should be noted that the machine learning models involved in one or more embodiments of this specification include, but are not limited to, fitting of one or more of the following machine learning models: Convolutional Neural Network (CNN) model, Deep Neural Network (DNN) model, Recurrent Neural Networks (RNN) model, embedding model, Gradient Boosting Decision Tree (GBDT) model, Logistic Regression (LR) model, etc.

[0143] S406: Use the validation group to validate the effect of the initial internal table demand prediction model to obtain the trained internal table demand prediction model.

[0144] S406 can be interspersed within the iterative training loop of step S404. A fixed evaluation period is set (e.g., every epoch or every 100 batches), the gradient update operation on the model weights is paused, and the model is switched to inference mode. The feature vectors in the validation group are batch-input into the initial internal table demand prediction model of the current state to obtain the prediction result (usually a prediction probability value). Based on the prediction result and the true labels of the validation group, key performance indicators are calculated. For example, AUC (Area Under Curve) can be selected as the core evaluation criterion because it has good robustness to positive and negative sample imbalance, and at the same time, it helps to monitor the changing trend of log loss (LogLoss).

[0145] Furthermore, a sliding window-based convergence determination logic is executed. A register recording the historical best metrics is maintained in advance, and the evaluation metric calculated in the current round is compared with the historical records. If the current metric is better than the historical records, the system temporarily stores the current weight parameters and resets the counter. If the current metric does not show a significant improvement within a preset number of training rounds (e.g., an improvement of less than 1e-4), or the loss value on the validation set begins to show an upward trend, it is determined that the model has reached convergence or a performance bottleneck has been reached. At this time, a termination command is automatically triggered to stop the training process. The last temporarily stored optimal weight parameter combination is read, serialized into a binary model file or a standard PMML file, and marked as a trained internal table demand prediction model. Subsequently, it is deployed to the online prediction service node to execute the real-time inference task in step S104.

[0146] In one feasible implementation, the first storage system includes a partitioned storage structure, please refer to [reference needed]. Figure 5 , Figure 5 This is a flowchart illustrating a data migration process. Specifically, the automated data migration task synchronizes the data entity of the target object to be accelerated from the first storage system to the second storage system, including:

[0147] S502: Identify the recommended data partition with the highest predicted acceleration gain value among the target objects to be accelerated;

[0148] Partitioned storage structure refers to a data organization method adopted by the primary storage system, in which the data of a large table is physically distributed and stored in different directories or file blocks according to specific key values. Each partition is an independent subdirectory in the file system.

[0149] Recommended data partitions refer to a specific set of partitions identified as having high access frequency and extremely high acceleration cost-effectiveness among all physical partitions of the target object. For example, for a log table partitioned by day, the partitions from the most recent 3 days are usually identified as recommended data partitions.

[0150] To illustrate, the system retrieves a list of partition metadata for the target object to be accelerated, iterates through all physical partition directories in the first storage system, and performs micro-level value prediction for each individual partition. Specifically, the aforementioned access value prediction model can be reused, using the physical size of each partition, the frequency of its appearance as a filter condition in historical queries, and the corresponding time attribute of the partition (e.g., partition key value dt=20231024) as input features to calculate the local acceleration gain prediction value for each partition. Alternatively, the system can use a time-decay-based heuristic rule to directly mark the N partitions closest to the current time (e.g., the last 7 days) or the K partitions ranked by access popularity as recommended data partitions with the highest acceleration gain prediction value. This process can accurately identify the data blocks that are truly frequently accessed in massive amounts of data, eliminating a large number of historically neglected partitions.

[0151] S504: Synchronize the data entities of the recommended data partition to the second storage system, and retain other partition data of the target object to be accelerated in the first storage system.

[0152] Executing S504 in this specification is equivalent to a hybrid storage state: it refers to the state in which the data entities of the same logical data table are physically separated and stored in two heterogeneous systems: hot data is located in the second storage system (high performance), and cold data is located in the first storage system (low cost).

[0153] As an illustration, for hot data identified as part of the recommended data partition, the system generates a targeted data synchronization task to load the binary data stream of that specific partition into the internal storage table of the second storage system via a high-speed network channel. For data in other partitions that are not selected (cold data), the system performs no migration operation, allowing it to remain in the first storage system.

[0154] Furthermore, to ensure transparency for upper-layer business queries, the system constructs a heterogeneous hybrid view in the second storage system or utilizes the database engine's intelligent routing capabilities. The view's metadata definition includes partition mapping logic: when the query request's WHERE condition hits the recommended data partition (hot), the query engine directly scans the local high-speed internal tables; when the query request hits other partitions (cold), the query engine automatically penetrates to the remote first storage system for reading via external mapping. In this way, the system achieves extreme acceleration for critical business hotspots with minimal storage and migration costs, without sacrificing data integrity, thus constructing a hybrid storage architecture that provides rapid response for hot data and fallback queries for cold data.

[0155] This specification employs a refined value assessment at the partition level and a differentiated data synchronization strategy (technical means) to effectively solve the technical problem of forced full migration of ultra-large-scale historical data tables due to the inability to separate hot and cold data, resulting in expensive high-performance storage resources being occupied by a large amount of low-frequency cold data and excessive data transmission bandwidth consumption (technical problem). This constructs a heterogeneous hybrid storage architecture that accelerates hot data locally and maps cold data remotely. While ensuring millisecond-level response for core business queries, it significantly reduces the hardware holding cost and data maintenance overhead of the second storage system.

[0156] In one feasible implementation, for S502-S504, the following steps are also included:

[0157] Step C2: Create a mapping table in the second storage system, the mapping table pointing to the full data in the first storage system;

[0158] A mapped table refers to a special database object defined in a secondary storage system. It only stores the table definition (table structure, column names, types) and path pointers to data files in the primary storage system (such as HDFS / OSS) in the secondary storage system, without actually storing the physical data content. When this table is queried, the system initiates a network request in real time to read the remote file.

[0159] Step C4: Configure a query routing strategy so that the query request response path for the recommended data partition is routed to the first storage system through the mapped external table in the accelerated storage unit of the second storage system, and the query request response path for the other partition data is routed to the first storage system through the mapped external table.

[0160] Query routing strategy refers to a set of logical rules configured by the database query optimizer or data virtualization layer. It can automatically determine whether to access the local high-speed accelerated storage unit or the remote mapped table based on the filtering conditions (especially the partition field) in the user's SQL statement.

[0161] Accelerated storage units refer to high-performance storage areas located in the local SSD or memory of the second storage system that have already synchronized the recommended data partition entity data in the previous steps.

[0162] To illustrate, the above steps are performed by the metadata management module of the data virtualization layer or database engine, aiming to build a hybrid cold and hot view transparent to upper-layer applications. The process involves cross-system schema mapping and connection establishment. Data definition language instructions are executed in the second storage system to create a mapping external table that is completely consistent with the structure of the target object to be accelerated. This operation establishes a metadata link pointing to the physical path of all data in the first storage system via the underlying network protocol, ensuring that the computing engine of the second storage system can directly read and parse the raw data files in the first storage system.

[0163] Subsequently, the system enters the core routing configuration and view construction phase. To achieve intelligent routing of query requests, a logically joined view covering local accelerated storage units (hot partitions) and mapped external tables (cold partitions) is built in the second storage system. During this process, query routing strategies are configured.

[0164] When a user initiates a query request, the SQL parser first extracts the partition key filter condition from the WHERE clause. If the condition matches a recommended data partition that has been synchronized to the local SSD (e.g., the last 7 days), the routing strategy will force the execution plan to scan only the local accelerated storage unit, thereby achieving millisecond-level fast response by utilizing columnar indexes and the vectorized engine. If the filter condition matches data from other unsynchronized partitions (e.g., historical data from 3 years ago), the routing strategy will automatically redirect the query operator to the mapped table and stream the data from the first storage system via network I / O. If the query range covers both hot and cold partitions, the execution engine will automatically split the task, retrieve data from both local and remote storage in parallel, and then aggregate it in memory. This mechanism allows upper-layer applications to achieve the hybrid storage advantages of extremely fast hot data and full cold data without modifying any code.

[0165] In practical applications, steps S502-S504 construct the underlying heterogeneous physical foundation, while steps C2-C4 construct the upper-level unified logical abstraction. Specifically, in implementation:

[0166] In one embodiment, the physical layer data migration described in steps S502 to S504 and the logical layer routing configuration described in steps C2 to C4 have a close temporal dependency and functional coupling relationship, together forming a complete transparent acceleration closed loop.

[0167] Specifically, its processing flow typically follows the following sequential logic:

[0168] First, in step C2, as an initialization step, the system establishes a mapping table in the second storage system that maintains a metadata link with the full data in the first storage system, thereby opening up the basic access channel between heterogeneous storage systems. At this time, all query requests are passed through this mapping table to the first storage system for execution by default, ensuring basic data accessibility.

[0169] Subsequently, the system enters the core optimization phase, executing steps S502 and S504. The system uses a predictive model to identify high-value recommended data partitions and physically loads their data entities into the local accelerated storage unit of the second storage system. At this point, two copies of the same data appear in the system: one is a full cold copy located in the first storage system (accessed via external table), and the other is a high-speed hot copy located in the second storage system (containing only the recommended partitions).

[0170] Finally, step C4 is executed as the service activation step. Based on the recommended data partition range locked in step S502, the system dynamically updates the query routing strategy or reconstructs the logical view. This strategy acts as a "smart switch" between physical data and upper-layer business, shielding the physical data sharding state caused by step S504 and ensuring that the query engine can automatically switch between the "local high-speed path" and the "remote fallback path" according to the latest data distribution. Through the above implementation sequence of first establishing logical connections (C2), then performing physical migration (S502-S504), and finally updating routing rules (C4), the system can smoothly realize the evolution of the storage architecture from single cold storage to a hybrid cold and hot tiered storage without the business being aware of it.

[0171] The following will combine Figure 6 This specification provides a detailed description of the data scheduling device provided in the embodiments. It should be noted that... Figure 6 The data scheduling device shown is used to execute this specification. Figures 1-5 The methods shown in the embodiments are illustrated for ease of explanation, showing only the parts related to the embodiments of this specification. For specific technical details not disclosed, please refer to this specification. Figures 1-5 The example shown.

[0172] Please see Figure 6 This diagram illustrates the structure of a data scheduling device according to an embodiment of this specification. The data scheduling device 1 can be implemented as all or part of a device through software, hardware, or a combination of both. According to some embodiments, the data scheduling device 1 includes an acquisition module 11, a prediction module 12, and a scheduling module 13, specifically used for:

[0173] The acquisition module 11 is used to acquire the metadata attribute characteristics and historical access statistics characteristics of candidate data objects in the first storage system;

[0174] Prediction module 12 is used to input the metadata attribute features and the historical access statistics features into a pre-trained internal table demand prediction model and output the acceleration gain prediction value corresponding to the candidate data table.

[0175] The scheduling module 13 is used to select target objects to be accelerated from the candidate data objects based on the acceleration gain prediction value, execute an automated data migration task to synchronize the data entities of the target objects to be accelerated from the first storage system to the second storage system, and establish an internal storage table corresponding to the target objects to be accelerated in the second storage system.

[0176] In one feasible implementation, the step of inputting the metadata attribute features and the historical access statistics features into a pre-trained internal table demand prediction model and outputting the acceleration gain prediction value corresponding to the candidate data table includes:

[0177] Based on the metadata attribute features, the physical storage capacity features of the candidate data objects are extracted, and based on the historical access statistics features, the data access frequency features of the candidate data objects under multiple time windows are statistically analyzed.

[0178] A basic dimension feature vector is generated based on the combination of the physical storage capacity characteristics and the data access frequency characteristics.

[0179] The basic dimension feature vector is input into a pre-trained internal table demand prediction model, which outputs an acceleration gain prediction value for the candidate data object.

[0180] In one feasible implementation, the plurality of time windows includes a short-term time window, a medium-term time window, and a long-term time window. The step of generating a basic dimension feature vector based on the combination of the physical storage capacity characteristics and the data access frequency characteristics includes:

[0181] The physical storage capacity characteristics and the data access frequency characteristics are normalized to construct a time-series heat characteristic group including short-term burst degree, medium-term stability degree and long-term decline degree. Based on the physical storage capacity characteristics, the access density characteristics per unit storage cost are determined.

[0182] Determine the historical query logs corresponding to the historical access statistical features, determine the computational operator type and data scan volume corresponding to the structured query statements in the historical query logs, determine the query complexity score based on the computational operator type and the data scan volume, and use the query complexity score as a computational cost dimension feature.

[0183] Based on the time-series heat feature group, the access density feature, the computation cost dimension feature, the physical storage capacity feature, and the data access frequency feature, feature engineering is performed to obtain the basic dimension feature vector.

[0184] In one feasible implementation, the step of inputting the basic dimension feature vector into a pre-trained internal table demand prediction model and outputting an acceleration gain prediction value for the candidate data object includes:

[0185] The basic dimension feature vectors are input into the pre-trained internal table demand prediction model;

[0186] Based on the computational cost dimension feature and data access frequency feature in the basic dimension feature vector, the internal table demand prediction model infers the cumulative query saving computational time after migrating the candidate data object to the second storage system. Based on the time series heat feature group, the future trend correction coefficient for the cumulative query saving computational time is determined. Based on the physical storage capacity feature and the access density feature, a resource occupation penalty value representing the migration cost is generated.

[0187] The positive resource benefit is obtained by correcting the cumulative query time saving calculation time using the future trend correction coefficient through the internal table demand prediction model. Based on the positive resource benefit and the resource occupation penalty, the acceleration gain prediction value is obtained, and the acceleration gain prediction value for the candidate data object is output.

[0188] In one feasible implementation, the method further includes:

[0189] Periodically synchronize the reference metadata attribute characteristics and reference historical access statistics characteristics of existing reference internal storage tables from the system tables of the second storage system;

[0190] The reference metadata attribute features and reference historical access statistics features are input into the internal table demand prediction model to perform a survival evaluation on the reference internal storage table and calculate the survival necessity probability value.

[0191] If the probability value of continued existence indicates that acceleration is not required, then an automated cleanup task is performed to delete the corresponding reference internal storage table from the second storage system.

[0192] In one feasible implementation, the internal demand forecasting model is trained using the following steps:

[0193] Select the data tables that are currently configured as internal storage tables in the second storage system as positive samples, and randomly select a preset proportion of data tables from the data tables that are not configured as internal storage tables in the first storage system as negative samples;

[0194] Positive and negative samples are divided into training and validation groups. The training group is used to train the initial internal table demand prediction model for at least one round to establish a linear mapping relationship between the input data and the internal table demand label. The output of the linear mapping relationship is mapped to a probability value using the model activation function, and the probability value is defined as the acceleration gain prediction value.

[0195] The initial internal table demand prediction model is validated using the validation set to obtain a trained internal table demand prediction model.

[0196] In one feasible implementation, the first storage system includes a partitioned storage structure, and the step of performing an automated data migration task to synchronize the data entity of the target object to be accelerated from the first storage system to the second storage system includes:

[0197] Identify the recommended data partition with the highest predicted acceleration gain among the target objects to be accelerated;

[0198] The data entities of the recommended data partition are synchronized to the second storage system, and the other partition data of the target object to be accelerated are retained in the first storage system.

[0199] In one feasible implementation, the method further includes:

[0200] Create a mapping table in the second storage system, the mapping table pointing to the full data in the first storage system;

[0201] Configure a query routing strategy so that the query request response path for the recommended data partition is routed to the first storage system through the mapping table in the accelerated storage unit of the second storage system, and the query request response path for the other partition data is routed to the first storage system through the mapping table.

[0202] It should be noted that the data scheduling device provided in the above embodiments is only illustrated by the division of the above functional modules when executing the data scheduling method. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the data scheduling device and the data scheduling method embodiments provided in the above embodiments belong to the same concept, and the implementation process can be found in the method embodiments, which will not be repeated here.

[0203] The example numbers in this specification are for descriptive purposes only and do not represent the superiority or inferiority of the examples.

[0204] This specification also provides a computer storage medium that can store multiple instructions adapted to be loaded and executed by a processor as described above. Figures 1-5 The data scheduling method described in the illustrated embodiment can be found in the following document for a detailed execution process. Figures 1-5 The specific details of the illustrated embodiments will not be elaborated here.

[0205] This specification also provides a computer program product that stores at least one instruction, said at least one instruction being loaded and executed by the processor as described above. Figures 1-5 The data scheduling method described in the illustrated embodiment can be found in the following document for a detailed execution process. Figures 1-5 The specific details of the illustrated embodiments will not be elaborated here.

[0206] Please refer to Figure 7 This is a structural block diagram of an electronic device provided in an embodiment of this specification. The electronic device in this specification may include one or more of the following components: a processor 1010, a memory 1020, an input device 1030, an output device 1040, and a bus 1050. The processor 1010, memory 1020, input device 1030, and output device 1040 may be connected to each other via the bus 1050.

[0207] Processor 1010 may include one or more processing cores. Processor 1010 connects to various parts of the electronic device using various interfaces and lines, and performs various functions and processes data by running or executing instructions, programs, code sets, or instruction sets stored in memory 1020, and by calling data stored in memory 1020. Optionally, processor 1010 may be implemented using at least one hardware form of digital signal processing (DSP), field-programmable gate array (FPGA), or programmable logic array (PLA). Processor 1010 may integrate one or more of a central processing unit (CPU), graphics processing unit (GPU), and modem. The CPU primarily handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the displayed content; and the modem handles wireless communication. It is understood that the modem may also not be integrated into processor 1010 and may be implemented separately through a communication chip.

[0208] The memory 1020 may include random access memory (RAM) or read-only memory (ROM). Optionally, the memory 1020 may include non-transitory computer-readable storage medium. The memory 1020 may be used to store instructions, programs, code, code sets, or instruction sets.

[0209] The input device 1030 is used to receive input instructions or data, and includes, but is not limited to, a keyboard, mouse, camera, microphone, or touch device. The output device 1040 is used to output instructions or data, and includes, but is not limited to, a display device and a speaker. In this embodiment, the input device 1030 can be a temperature sensor for acquiring the operating temperature of the electronic device. The output device 1040 can be a speaker for outputting audio signals.

[0210] In addition, those skilled in the art will understand that the structure of the electronic device shown in the above figures does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown, or combine certain components, or have different component arrangements. For example, the electronic device may also include radio frequency circuits, input units, sensors, audio circuits, wireless fidelity (WIFI) modules, power supplies, Bluetooth modules, etc., which will not be described in detail here.

[0211] In the embodiments of this specification, the executing entity for each step can be the electronic device described above. Optionally, the executing entity for each step can be the operating system of the electronic device. The operating system can be Android, iOS, or other operating systems; this specification does not limit this.

[0212] exist Figure 7 In the electronic device, the processor 1010 can be used to call a program stored in the memory 1020 and execute it to implement the data scheduling method as described in the various method embodiments of this specification.

[0213] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory, or random access memory, etc.

[0214] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.), and signals involved in the embodiments of this specification are all authorized by the user or fully authorized by all parties, and the collection, use, and processing of related data must comply with the relevant laws, regulations, and standards of the relevant countries and regions. For example, the candidate data objects involved in this specification were all obtained under full authorization.

[0215] The above-disclosed embodiments are merely preferred embodiments of this specification and should not be construed as limiting the scope of this specification. Therefore, any equivalent variations made in accordance with the claims of this specification shall still fall within the scope of this specification.

Claims

1. A data scheduling method, characterized in that, Applied to a data processing platform including a first storage system and a second storage system, wherein the query response latency of the second storage system is lower than that of the first storage system, the method includes: Obtain the metadata attribute characteristics and historical access statistics of candidate data objects in the first storage system; Based on the metadata attribute features, the physical storage capacity features of the candidate data objects are extracted; based on the historical access statistics features, the data access frequency features of the candidate data objects under multiple time windows are statistically analyzed; and based on the physical storage capacity features and the data access frequency features, a basic dimension feature vector is generated. The basic dimension feature vector is input into a pre-trained internal table demand prediction model. Based on the computational cost dimension feature and data access frequency feature in the basic dimension feature vector, the model infers the cumulative query saving computational time after migrating the candidate data object to the second storage system. Based on the time-series heat feature group in the basic dimension feature vector, the model determines the future trend correction coefficient for the cumulative query saving computational time. Based on the physical storage capacity feature and the access density feature in the basic dimension feature vector, the model generates a resource occupation penalty value that represents the migration cost. The positive resource benefit is obtained by correcting the cumulative query saving calculation time through the internal table demand prediction model using the future trend correction coefficient. Based on the positive resource benefit and the resource occupation penalty, the acceleration gain prediction value is obtained, and the acceleration gain prediction value for the candidate data object is output. Based on the predicted acceleration gain value, the target objects to be accelerated are selected from the candidate data objects; An automated data migration task is performed to synchronize the data entity of the target object to be accelerated from the first storage system to the second storage system, and an internal storage table corresponding to the target object to be accelerated is established in the second storage system.

2. The method according to claim 1, characterized in that, The multiple time windows include short-term time windows, medium-term time windows, and long-term time windows. The generation of a basic dimension feature vector based on the combination of the physical storage capacity characteristics and the data access frequency characteristics includes: The physical storage capacity characteristics and the data access frequency characteristics are normalized to construct a time-series heat characteristic group including short-term burst degree, medium-term stability degree and long-term decline degree. Based on the physical storage capacity characteristics, the access density characteristics per unit storage cost are determined. Determine the historical query logs corresponding to the historical access statistical features, determine the computational operator type and data scan volume corresponding to the structured query statements in the historical query logs, determine the query complexity score based on the computational operator type and the data scan volume, and use the query complexity score as a computational cost dimension feature. Based on the time-series heat feature group, the access density feature, the computation cost dimension feature, the physical storage capacity feature, and the data access frequency feature, feature engineering is performed to obtain the basic dimension feature vector.

3. The method according to claim 1, characterized in that, The method further includes: Periodically synchronize the reference metadata attribute characteristics and reference historical access statistics characteristics of existing reference internal storage tables from the system tables of the second storage system; The reference metadata attribute features and reference historical access statistics features are input into the internal table demand prediction model to perform a survival evaluation on the reference internal storage table and calculate the survival necessity probability value. If the probability value of continued existence indicates that acceleration is not required, then an automated cleanup task is performed to delete the corresponding reference internal storage table from the second storage system.

4. The method according to any one of claims 1-3, characterized in that, The internal demand forecasting model is trained using the following steps: Select the data tables that are currently configured as internal storage tables in the second storage system as positive samples, and randomly select a preset proportion of data tables from the data tables that are not configured as internal storage tables in the first storage system as negative samples; Positive and negative samples are divided into training and validation groups. The training group is used to train the initial internal table demand prediction model for at least one round to establish a linear mapping relationship between the input data and the internal table demand label. The output of the linear mapping relationship is mapped to a probability value using the model activation function, and the probability value is defined as the acceleration gain prediction value. The initial internal table demand prediction model is validated using the validation set to obtain a trained internal table demand prediction model.

5. The method according to claim 1, characterized in that, The first storage system includes a partitioned storage structure. The step of executing an automated data migration task to synchronize the data entity of the target object to be accelerated from the first storage system to the second storage system includes: Identify the recommended data partition with the highest predicted acceleration gain among the target objects to be accelerated; The data entities of the recommended data partition are synchronized to the second storage system, and the other partition data of the target object to be accelerated are retained in the first storage system.

6. The method according to claim 5, characterized in that, The method further includes: Create a mapping table in the second storage system, the mapping table pointing to the full data in the first storage system; Configure a query routing strategy so that the query request response path for the recommended data partition is routed to the first storage system through the mapping table in the accelerated storage unit of the second storage system, and the query request response path for the other partition data is routed to the first storage system through the mapping table.

7. A data scheduling method apparatus, characterized in that, The method apparatus is applied to a data processing platform comprising a first storage system and a second storage system, wherein the query response latency of the second storage system is lower than that of the first storage system. The acquisition module is used to acquire the metadata attribute characteristics and historical access statistics characteristics of candidate data objects in the first storage system; The prediction module is used to extract the physical storage capacity features of the candidate data objects based on the metadata attribute features, to calculate the data access frequency features of the candidate data objects under multiple time windows based on the historical access statistics features, to generate a basic dimension feature vector based on the combination of the physical storage capacity features and the data access frequency features, to input the basic dimension feature vector into a pre-trained internal table demand prediction model, and to infer the cumulative query saving computation time after migrating the candidate data objects to the second storage system based on the computation cost dimension features and data access frequency features in the basic dimension feature vector, to determine the future trend correction coefficient for the cumulative query saving computation time based on the time-series heat feature group in the basic dimension feature vector, and to generate a resource occupation penalty value representing the migration cost based on the physical storage capacity features and the access density features in the basic dimension feature vector; to obtain positive resource benefits by correcting the cumulative query saving computation time through the internal table demand prediction model using the future trend correction coefficient, to obtain an acceleration gain prediction value based on the positive resource benefits and the resource occupation penalty value, and to output the acceleration gain prediction value for the candidate data objects. The scheduling module is used to select target objects to be accelerated from the candidate data objects based on the acceleration gain prediction value, execute an automated data migration task to synchronize the data entities of the target objects to be accelerated from the first storage system to the second storage system, and establish an internal storage table corresponding to the target objects to be accelerated in the second storage system.

8. A computer storage medium, characterized in that, The computer storage medium stores a plurality of instructions, which are adapted to be loaded by a processor and executed as described in any one of claims 1 to 6.

9. A computer program product, characterized in that, The computer program product stores at least one instruction, which is loaded by a processor and executed as described in any one of claims 1 to 6.

10. An electronic device, characterized in that, include: A processor and a memory; wherein the memory stores a computer program adapted to be loaded by the processor and executed as described in any one of claims 1 to 6.

Images