Data table storage method and device and related equipment

Abstract

The invention provides a data table storage method and device and related equipment, and relates to the technical field of computers.The method comprises the steps that accessed access data in at least one data table within a first preset time period is obtained; training a first model based on the accessed access data in the at least one data table to obtain a second model; the accessed condition of each data table is predicted through the second model, and the predicted accessed probability corresponding to each data table output by the second model is obtained; and based on the predicted accessed probability corresponding to each data table, determining a storage medium corresponding to each data table.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a data table storage method, apparatus and related equipment. Background Technology

[0002] With the development of internet technology, data is also growing exponentially. Data tables within this data can be stored in remote storage systems. These systems typically feature tiered storage, with each tier capable of using different types of storage media, each differing significantly in price and read / write speed. Electronic devices can then store data tables on different storage media based on their access patterns.

[0003] However, in related technologies, electronic devices cannot sense or accurately sense the access status of data tables. Consequently, when the access status of data tables changes continuously, electronic devices cannot adjust the storage medium on which the data tables are stored in real time according to the changes in access status. This can easily lead to improper placement of data tables, resulting in a waste of storage resources. Summary of the Invention

[0004] This application provides a data table storage method, apparatus, and related equipment, which can predict the access status of the data table in advance and adjust the storage medium of the data table in a timely manner, so that the data table can be stored in a suitable storage medium and avoid the waste of storage resources.

[0005] In a first aspect, embodiments of this application provide a data table storage method, the method comprising: acquiring access data accessed in at least one data table within a first preset time period; The first model is trained based on the access data accessed in at least one of the data tables to obtain the second model; The second model is used to predict the access status of each data table, and the predicted access probability of each data table is obtained from the output of the second model. Based on the predicted access probability of each data table, the storage medium corresponding to each data table is determined.

[0006] Optionally, the access data includes at least one of the following: the table size corresponding to the data table, the number of rows occupied by the data table, the number of columns occupied by the data table, the data table creation time interval, at least one access time interval, and whether the data table was accessed within the first preset time period, wherein the data table creation time interval is the difference between the training time point corresponding to the first model and the data table creation time point, and the access time interval is the difference between two adjacent access timestamps of the data table.

[0007] Optionally, training the first model based on access data accessed in the at least one data table to obtain the second model includes: The first model is trained based on the access data accessed in the at least one data table, and the predicted access status of each data table in the at least one data table output by the first model within a preset time window is obtained. The accuracy of the first model is obtained by comparing the actual access status of each data table in the at least one data table within the preset time window with the predicted access status. If the accuracy of the first model is less than or equal to the first preset threshold, the step of training the first model based on the access data accessed in the at least one data table continues until the accuracy is greater than the first preset threshold, so as to obtain the second model.

[0008] Optionally, determining the storage medium corresponding to each data table based on the predicted access probability of each data table includes: Based on the predicted access probability of each data table and the table size of each data table, the storage score of each data table is determined. Data tables whose storage scores are greater than the second preset threshold are stored in the first storage medium, and data tables whose storage scores are less than or equal to the second preset threshold are stored in the second storage medium.

[0009] Optionally, determining the storage score for each data table based on the predicted access probability and the table size of each data table includes: Based on the sum of the table sizes of all data tables and the table size corresponding to each data table, the normalized table size corresponding to each data table is obtained. The storage score for each data table is obtained by weighted summing of the table size corresponding to each normalized data table and the predicted access probability corresponding to each data table.

[0010] Optionally, the first storage medium includes a memory pool and a cloud disk, and the second storage medium includes a hard disk; storing data tables with storage scores greater than a second preset threshold to the first storage medium, and storing data tables with storage scores less than or equal to the second preset threshold to the second storage medium, includes: Data tables whose storage scores are greater than the second preset threshold but less than or equal to the third preset threshold are stored in the cloud disk; the third preset threshold is greater than the second preset threshold. Store the data tables whose storage scores are greater than the third preset threshold into the memory pool; Data tables whose storage scores are less than or equal to the second preset threshold are stored on the hard disk.

[0011] Optionally, the method further includes: Calculate the correlation between the data tables in the first data table set and the data tables in the second data table set; the data tables in the first data table set are stored in the first storage medium, and the data tables in the second data table set are stored in the second storage medium; The data tables in the second set of data tables whose relevance is greater than the fourth preset threshold are determined as target data tables; The target data table is stored in the first storage medium.

[0012] Optionally, calculating the correlation between data tables in the first data table set and data tables in the second data table set includes: Get the total number of times each data table in the first data table set is accessed within the second preset time period, the total number of times each data table in the second data table set is accessed within the second preset time period, and the number of times the first data table and the second data table are accessed simultaneously within the second preset time period; the first data table is a data table in the first data table set, and the second data table is a data table in the second data table set; The relevance is obtained based on the total number of times each data table in the first data table set is accessed, the total number of times each data table in the second data table set is accessed, and the number of times the first data table and the second data table are accessed simultaneously.

[0013] Secondly, embodiments of this application provide a data table storage device, including: The acquisition module is used to acquire access data from at least one data table within a first preset time period. The training module is used to train the first model based on the access data accessed in the at least one data table to obtain the second model; The prediction module is used to predict the access status of each data table through the second model and obtain the predicted access probability of each data table output by the second model. The first determining module is used to determine the storage medium corresponding to each data table based on the predicted access probability of each data table.

[0014] Thirdly, embodiments of this application provide an electronic device, including: a processor, a memory, and a program stored in the memory and executable on the processor, wherein when the program is executed by the processor, it implements the steps of the data table storage method as described in the first aspect.

[0015] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the data table storage method as described in the first aspect.

[0016] Fifthly, embodiments of this application provide a computer program product, including computer instructions that, when executed by a processor, implement the steps of the data table storage method as described in the first aspect.

[0017] In this embodiment, the electronic device can train a first model using access data from at least one data table accessed within a first preset time period to obtain a second model. The electronic device then uses the second model to predict the access status of each data table within the at least one data table and obtains the predicted access probability for each data table output by the second model. The electronic device then determines the corresponding storage medium for each data table based on the predicted access probability. In this process, the second model in the electronic device is obtained by training the first model using access data from at least one data table accessed within the first preset time period. Therefore, the second model obtained by the electronic device can be effectively trained with high accuracy. Consequently, the accuracy of the predicted access probability for each data table output by the second model is also high. The electronic device can also more accurately determine the corresponding storage medium for each data table based on the predicted access probability. This allows the electronic device to predict the access status of data tables in advance and accurately, enabling the electronic device to adjust the storage medium of the data tables in a timely manner, ensuring that the data tables are stored in suitable storage media and avoiding waste of storage resources. Attached Figure Description

[0018] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a flowchart of a data table storage method provided in an embodiment of this application; Figure 2 This is one of the timing diagrams of a data table being accessed provided in the embodiments of this application; Figure 3 This is a second timing diagram illustrating the access of a data table provided in an embodiment of this application; Figure 4 This is the third time-series diagram illustrating the access of a data table provided in this application embodiment; Figure 5 This is a flowchart of a first model training method provided in an embodiment of this application; Figure 6 This is a schematic diagram of the structure of a data table storage device provided in an embodiment of this application; Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0020] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0021] With the continuous development of key technologies such as cloud computing and the Internet of Things, the scale, variety, and speed of data production are experiencing explosive growth. In a cloud environment, data tables are typically stored in remote storage systems. Compared to local environments, cloud-based remote storage systems usually encompass a wider range of storage media, such as regular cloud disks, high-performance cloud disks, solid-state drives (SSDs), and hard disk drives (HDDs). These remote storage systems often feature tiered storage, with each tier using different types of storage media. However, different storage media vary significantly in price and read / write speeds, necessitating the selection of appropriate storage media for data tables.

[0022] However, the query activity of a data table changes in real time, leading to a constant fluctuation in the table's access frequency, or its popularity. In related technologies, electronic devices cannot detect or accurately perceive the access status of a data table. Consequently, when the access status changes continuously, the electronic device cannot adjust the storage medium for the data table in real time, potentially resulting in improper data table placement and wasted storage resources.

[0023] For example, related technologies can be broadly categorized into two types. The first type involves electronic devices using historical data to train deep learning or machine learning models, predicting future access patterns, and then using this prediction to select cache eviction strategies, data storage media, or data partitioning. The second type uses heuristic methods to classify data into hot and cold categories based on factors such as access frequency and node load, and then uses set hot and cold thresholds to determine and trigger data migration. However, neither of these types of technologies can store data tables in suitable storage media, easily leading to a waste of storage resources.

[0024] Therefore, this application provides a data table storage method that not only solves the problems existing in related technologies, but also predicts the access status of the data table in advance and adjusts the storage medium of the data table in a timely manner, so that the data table can be stored in a suitable storage medium and avoids the waste of storage resources.

[0025] See Figure 1 , Figure 1 This is a flowchart of a data table storage method provided in an embodiment of this application, such as... Figure 1 As shown, the method includes the following steps: Step 101: Obtain access data from at least one data table within the first preset time period.

[0026] In some embodiments, the first preset time period is a time period preceding the current time. Therefore, the access data acquired by the electronic device within the first time period is the access data accessed in at least one data table in real time. This provides real-time training data for subsequent model training, ensuring that the real-time trained model can accurately generate predictions of access results.

[0027] In some embodiments, access data is obtained based on the access request (or query request) of the data table and the metadata of the data table.

[0028] Optionally, the electronic device can monitor the access logs (or query logs) for each data table in real time, obtaining access requests and metadata for each data table through the access logs. For example, the electronic device can use Kafka or other messaging components as message producers to generate real-time data streams of data table accesses, while simultaneously collecting metadata for each data table and storing it in the database.

[0029] The access request for a data table is a stream of accessed data generated in real time based on the access log. The access request includes: a set of unique identifiers for the data table and the access timestamp of the data table. Since multiple data tables may be joined at the same time, the access request includes the set of unique identifiers for the data tables, and the access timestamp is the timestamp when the data table was currently queried. The metadata of the data table includes: the unique identifier of the data table, the size of the corresponding table, the number of rows in the table, the number of columns in the table, and the current storage medium of the data table.

[0030] After the electronic device obtains the access request and metadata of the data table, it can integrate the access request and metadata to obtain integrated metadata, which can provide real-time training data for subsequent model training.

[0031] Optionally, electronic devices can use streaming data processing modules such as Flink to integrate access requests and metadata of the data table, generating new integrated metadata for the data table. This integrated metadata includes: a unique identifier for the data table, the table size, the number of rows, the number of columns, the current storage medium, the table creation time, and the timestamp of when the data table was first accessed. The table creation time is the timestamp when the data table was first created.

[0032] After obtaining the integrated metadata, the electronic device can preprocess it to obtain access data, which is then used as training data to train the model. For example, the electronic device can generate feature vectors based on the access data, then normalize the feature vectors, and use the normalized feature vectors as training data for the model. The electronic device can also obtain access data through other data acquisition methods; no restrictions are placed on this.

[0033] In some embodiments, the accessed data includes at least one of the following: the table size corresponding to the data table, the number of rows occupied by the data table, the number of columns occupied by the data table, the data table creation time interval, at least one access time interval, and whether the data table is accessed within the first preset time period, wherein the data table creation time interval is the difference between the training time point corresponding to the first model and the data table creation time point, and the access time interval is the difference between two adjacent access timestamps of the data table.

[0034] In this embodiment, assuming that model training is triggered at the current time point, the electronic device can use the current time point as the training time point and obtain access data of at least one data table in a time period preceding the current time point. That is, it can obtain access data of at least one data table in a preset time period and use the access data as training data to train the model.

[0035] The data table creation time interval refers to the current training time point. The interval between the creation time and the table's creation time. Since data tables are frequently used in the initial stages of creation, and their usage frequency decreases significantly over time, the creation time interval can be used to reflect the probability of the data table being accessed.

[0036] The access time interval refers to the difference between two consecutive access timestamps of the data table. There can be k access time intervals, where k is an integer greater than 1. k is a pre-set number of access time intervals, meaning the electronic device only considers the most recent k adjacent data access time intervals. The electronic device will obtain the most recent k access time intervals as access data, which reflects the access status of the data table. A larger k value indicates that the model will learn data access patterns over a longer period, while a smaller k value indicates that the model will learn data access patterns over a shorter period. k is usually not set to an excessively large value. For newly created data tables or data tables accessed less than k times, the generated access data cannot provide sufficient information for the model. These data tables will be considered cold data, i.e., data tables with low access frequency, and the electronic device will temporarily store these data tables in low-performance storage media. In other words, when the access frequency of a data table is less than k, the data table does not participate in the prediction update of access probability or the migration of the data table.

[0037] Electronic devices can use 1 or 0 to indicate whether a data table has been accessed within a first preset time period. For example, there is a point in time within the first preset time period. The data table is in If the data table is accessed within the first preset time period, the access data value is 1, meaning the data table was accessed within that period. Conversely, if the data table is not accessed within the first preset time period, the access data value is 0. Alternatively, if the data table is not accessed within the first preset time period... , making , window It is the preset time window size. That is, if the data table is accessed within the preset time window of the first preset time period, the accessed data is 1, otherwise the accessed data is 0.

[0038] Understandably, the access data in this embodiment can provide real-time training data for subsequent model training, ensuring that the real-time trained model can accurately generate predictions of the accessed results. Furthermore, the access data fully considers the characteristics of the data table and the data access status, including the table size, the number of rows and columns, the table creation time interval, at least one access time interval, whether the data table was accessed within a first preset time period, and the current storage medium of the data table. These features can effectively improve the generalization ability during subsequent model training.

[0039] Step 102: Train the first model based on the access data accessed in the at least one data table to obtain the second model.

[0040] In the embodiments of this application, the model can be an ensemble learning model, such as a prediction model using a lightweight gradient boosting machine. The lightweight gradient boosting machine is an efficient implementation of a gradient boosting decision tree framework. Compared to other ensemble learning models, it has advantages in handling large-scale data and training speed. The lightweight gradient boosting machine employs a histogram-based algorithm to discretize feature values, transforming continuous features into histogram buckets. This reduces computational load and memory consumption, improving training speed. Furthermore, the lightweight gradient boosting machine uses column-oriented data storage, further saving memory and using less memory during training. Compared to deep learning models (such as neural network algorithms), it is lightweight and more suitable for online training and prediction scenarios. Simultaneously, compared to traditional data popularity calculation methods based on formulas (such as decay functions), the lightweight gradient boosting machine is better able to capture the non-linear relationship between data features and data popularity, thereby achieving more accurate predictions of future access to the data table. Therefore, lightweight gradient boosting machines can be widely used in scenarios suitable for processing large-scale training data, high-dimensional data training, and real-time prediction, and the embodiments of this application also preferably use the prediction model of lightweight gradient boosting machines.

[0041] In some embodiments, electronic devices may also use deep learning models, but ensemble learning models are preferred.

[0042] Furthermore, the predictive models in these technologies suffer from poor real-time training performance. Specifically, the machine learning methods rely on large amounts of historical data for training. However, in real-world scenarios, data access patterns change dynamically. To ensure accurate predictions, these technologies require stopping the running model and collecting historical data for retraining; otherwise, the model's accuracy may not meet expectations. Moreover, these technologies cannot perform online training and evaluation of the predictive model during its operation.

[0043] Optionally, in this embodiment of the application, training the first model based on the access data accessed in the at least one data table to obtain the second model includes: The first model is trained based on the access data accessed in the at least one data table, and the predicted access status of each data table in the at least one data table output by the first model within a preset time window is obtained. The accuracy of the first model is obtained by comparing the actual access status of each data table in the at least one data table within the preset time window with the predicted access status. If the accuracy of the first model is less than or equal to the first preset threshold, the step of training the first model based on the access data accessed in the at least one data table continues until the accuracy is greater than the first preset threshold, so as to obtain the second model.

[0044] Understandably, in this embodiment, to address the difficulty in obtaining large amounts of log data for offline training and the problem of weak model timeliness caused by training with historical access data, the electronic device employs a real-time training method. During the operation of the electronic device, it generates access data corresponding to at least one data table used for training in real time. This access data is used to dynamically perform incremental training on the first model, eliminating the need to collect large amounts of historical access data for offline training and ensuring strong model timeliness. Furthermore, during the operation of the electronic device, the timing for completing the training of the first model is determined by calculating the accuracy. After training, if the accuracy is lower than a first preset threshold, training and evaluation of the first model can continue, allowing the trained model to adapt to new data access scenarios.

[0045] For example, suppose the current training time point is... The first preset time period is Electronic devices can Access data from at least one of the data tables is input into the first model. And... After the time-point training is completed, the predicted access status of each data table in at least one data table output by the first model within a preset time window is obtained.

[0046] Optionally, the electronic device can be used in the next time period. Randomly select several time points And for each time point Set the corresponding preset time window. That is, each time point. The corresponding preset time window is And these preset time windows are within the time period Inside.

[0047] The electronic device then iterates through at least one data table, using a first model to predict whether new requests will be made to access the data table within each preset time window. For example, ... Figure 2 As shown, assume that the prediction result of the first model is that there is a new request to access data table 1, that is, there is a time point within the preset time window. , At a future preset time window The data table 1 above was accessed. Conversely, if... Figure 3 As shown, assume that the prediction result of the first model is that there are no new requests to access data table 1, that is, there is no time point within the preset time window. This makes at a certain point in time The data table 1 above was accessed.

[0048] In some embodiments, the predicted access status of a data table within a preset time window is used to characterize whether the data table has been accessed within the preset time window. The first model outputs the access probability of the data table within the preset time window. Electronic devices can determine whether the data table has been accessed based on the access probability of the data table within the preset time window using a binary classification method. For example, assuming the access probability of the data table within the preset time window is greater than 0.5, it means that the data table has been accessed within the preset time window. Assuming the access probability of the data table within the preset time window is less than or equal to 0.5, it means that the data table has not been accessed within the preset time window.

[0049] Electronic devices can be reacquired within a time period The actual access status of each data table in at least one data table within a given time period; that is, the access status of each data table within a given time period. The system checks whether the data tables have actually been accessed. The electronic device then compares the actual access status of each data table with the predicted access status to obtain the average accuracy (or precision) of the first model. If the average accuracy is greater than a first preset threshold, training of the first model stops, resulting in the trained first model, which is the second model. Otherwise, if the average accuracy is less than or equal to the first preset threshold, the training stops. As a new training time point, Access data from at least one data table within the system is used as new training data to continue training the first model. During the subsequent operation of the electronic device, the average accuracy of the second model's predictions will continue to be calculated. If the average accuracy is less than or equal to a first preset threshold, it indicates that the access situation of the data tables has changed, and the second model cannot adapt to the new data access situation. At this time, the parameters of the second model are reset, and the second model is retrained according to the training method of the first model described above, until the average accuracy is greater than the first preset threshold.

[0050] One way for electronic devices to calculate average accuracy is to acquire the accuracy rates of each data table and then average these accuracy rates. For example, data table 1 within a given time period... , , All data points were accessed. The predicted access status of the first model output is shown in Table 1 during the time period. , The data was accessed, therefore the accuracy rate corresponding to Table 1 is... Similarly, data table 2 shows the time period... , , All of them were accessed. The access status of the prediction output of the first model on the electronic device is shown in Table 2 during the time period. , , The data was accessed, therefore the accuracy rate corresponding to Table 2 is... The electronic device calculates the average accuracy by averaging the accuracy rates corresponding to data table 1 and data table 2. The resulting average accuracy rate is... .

[0051] Alternatively, electronic devices can calculate average accuracy by acquiring the predicted and actual access data for each data table to determine the accuracy. For example, within a time period... Data tables 1, 2, and 3 were actually accessed. The predicted access status of the first model output on the electronic device was within the time period. Data tables 1 and 2 were accessed. Therefore, the accuracy rate is... .

[0052] The process of obtaining the second model from an electronic device will be explained in detail below through a specific example.

[0053] like Figure 4 As shown, at time point Trigger the training task. Assume k is 2. , This represents the time interval between two consecutive accesses to data. The triangle symbol indicates the point in time when the data table was created. As a preset time window, For reference time points, in and The data table has records accessed between the points in time; therefore, the number of accessed data is 1. for The creation time interval between the creation time of the data table and the creation time of the data table.

[0054] Therefore, accessing data can be represented as: the size of the data table, the number of rows in the data table, and the number of columns in the data table. , , 1. Similarly, access data in this format is generated for each data table for online training of the first model.

[0055] In the next time period Within, electronic devices are randomly sampled at several time points. And for each time point A corresponding preset time window is set. The electronic device iterates through each data table, using the first model to predict whether there will be new requests to access the data table within the preset time window. If the actual access situation matches the predicted access situation, it is recorded. The number that was correctly predicted. Then in Time period statistics The electronic device calculates the accuracy of the first model and its average accuracy. If the average accuracy exceeds a first preset threshold, the electronic device stops training the first model and obtains the trained first model, which is the second model. Otherwise, the electronic device continues training the first model until the average accuracy exceeds the first preset threshold, thus obtaining the second model.

[0056] like Figure 5 As shown, through Figure 5 The model training flowchart provides a detailed explanation of the model training process.

[0057] Step 501: Is the current time point within the training timeframe?

[0058] In some embodiments, if it is at a training time point, step 502 is executed. Otherwise, the determination of whether it is at a training time point continues.

[0059] Step 502: Obtain the predicted access status of each data table in at least one data table output by the first model within a preset time window.

[0060] Step 503: Based on the actual and predicted access status of each data table within the preset time window, obtain the accuracy of the first model.

[0061] Step 504: Is the accuracy of the first model less than or equal to the first preset threshold?

[0062] In some embodiments, if the accuracy of the first model is less than or equal to a first preset threshold, step 506 is executed; otherwise, step 505 is executed.

[0063] Step 505: Obtain the second model after training the first model.

[0064] Step 506: Continue training the first model.

[0065] Through steps 501-506 above, the electronic device can train the first model.

[0066] Step 103: Predict the access status of each data table using the second model, and obtain the predicted access probability of each data table output by the second model.

[0067] Step 104: Based on the predicted access probability of each data table, determine the storage medium corresponding to each data table.

[0068] In some embodiments, determining the storage medium corresponding to each data table based on the predicted access probability of each data table includes: Based on the predicted access probability of each data table and the table size of each data table, the storage score of each data table is determined. Data tables whose storage scores are greater than the second preset threshold are stored in the first storage medium, and data tables whose storage scores are less than or equal to the second preset threshold are stored in the second storage medium.

[0069] In this embodiment, the electronic device allocates the data table to a first storage medium or a second storage medium based on the storage score. The first storage medium can be a high-performance storage medium, and the second storage medium is a low-performance storage medium. The high-performance storage medium has higher read efficiency and throughput than the low-performance storage medium, and its latency is lower than that of the low-performance storage medium.

[0070] In this embodiment, the electronic device can determine the key features of the data table and thus obtain information about data access by using storage scores. There are various ways for the electronic device to calculate storage scores, and no limitation is made here.

[0071] Understandably, in this embodiment, the electronic device obtains the storage score of each data table by using the predicted access probability and table size corresponding to each data table. This allows the electronic device to accurately allocate each data table to a suitable storage medium based on the storage score, thus avoiding the waste of storage resources.

[0072] In some embodiments, determining the storage score for each data table based on the predicted access probability and the table size of each data table includes: Based on the sum of the table sizes of all data tables and the table size corresponding to each data table, the normalized table size corresponding to each data table is obtained. The storage score for each data table is obtained by weighted summing of the table size corresponding to each normalized data table and the predicted access probability corresponding to each data table.

[0073] In this embodiment, the formula for calculating the storage score can be: . The storage score corresponding to the data table. This represents the predicted probability of the data table being accessed. This refers to the table size corresponding to the normalized data table. and The weighting coefficients are between (0, 1). This indicates the impact of the predicted access probability of the corresponding data table on the storage score. This indicates the impact of the table size on the storage score. The weighting coefficients can be adjusted based on the specific application scenario. If it's desired that the probability of being accessed is the dominant factor, then the weighting coefficients can be adjusted. Set to a larger value; if the storage space of the high-performance storage medium is limited, and you want the high-performance storage medium to store as many data tables as possible to improve the storage hit rate, then you can adjust the weighting coefficient. Set to a larger value.

[0074] The process by which the electronic device calculates the size of the normalized data table can be as follows: The electronic device obtains the size of each data table, adds the sizes of each data table together to obtain the total size of all data tables. The electronic device then divides the size of the data table corresponding to each data table by the total size of all data tables to obtain the size of the normalized data table.

[0075] Understandably, in this embodiment, the electronic device can accurately obtain the storage score by weighted summing of the table size corresponding to each normalized data table and the predicted access probability corresponding to each data table, which facilitates the subsequent determination of the storage medium corresponding to each data table based on the storage score.

[0076] In some embodiments, the first storage medium includes a memory pool and a cloud disk, and the second storage medium includes a hard disk; storing data tables with storage scores greater than a second preset threshold to the first storage medium, and storing data tables with storage scores less than or equal to the second preset threshold to the second storage medium, includes: Data tables whose storage scores are greater than the second preset threshold but less than or equal to the third preset threshold are stored in the cloud disk; the third preset threshold is greater than the second preset threshold. Store the data tables whose storage scores are greater than the third preset threshold into the memory pool; Data tables whose storage scores are less than or equal to the second preset threshold are stored on the hard disk.

[0077] In this embodiment, the electronic device can further divide the storage media based on the first and second storage media to obtain a memory pool, a cloud disk, and a hard disk. The electronic device can set a second preset threshold for the memory pool, cloud disk, and hard disk. Assuming the second preset threshold is... The first preset threshold is .if If so, then the data table will be allocated to the memory pool; if If so, then the data table will be allocated to the cloud disk; if If so, then the data table will be allocated to the hard drive. Alternatively, storage media can be divided into four or more storage media types, and corresponding electronic devices can set preset thresholds for the storage media to allocate data tables; no restrictions are imposed here.

[0078] Understandably, in this embodiment, the electronic device achieves accurate storage of data tables by storing scores, enabling each data table to be allocated to a suitable storage medium and avoiding waste of storage resources.

[0079] Furthermore, during the storage media adjustment phase of related technologies, electronic devices did not consider the correlation and affinity between data tables. Specifically, the heuristic methods in related technologies mainly consider storage media adjustment strategies based on factors such as data access frequency, node load, and data hot / cold indices, ignoring the correlation and affinity between data. For example, suppose data table 1 is stored in a high-performance storage medium, making it easy to access, while data table 2 is stored in a low-performance storage medium, making it inconvenient to access. However, data table 1 and data table 2 have a high correlation; when an electronic device accesses data table 1, it will also access data table 2. But because data table 2 is stored in a low-performance storage medium, it is inconvenient to access. Therefore, placing data table 2 in an unsuitable storage medium affects its access efficiency.

[0080] Optionally, in embodiments of this application, the data table storage method further includes: Calculate the correlation between the data tables in the first data table set and the data tables in the second data table set; the data tables in the first data table set are stored in the first storage medium, and the data tables in the second data table set are stored in the second storage medium; The data tables in the second set of data tables whose relevance is greater than the fourth preset threshold are determined as target data tables; The target data table is stored in the first storage medium.

[0081] Understandably, in this embodiment, the electronic device determines the correlation between data tables in the first data table set and data tables in the second data table set, and identifies data tables in the second data table set with a correlation greater than a fourth preset threshold as target data tables. The electronic device then migrates the target data tables located in the second storage medium to the first storage medium. This ensures that multiple frequently accessed and highly correlated data tables are stored in the same first storage medium. When a frequently accessed data table is accessed, the probability of a highly correlated data table being accessed subsequently is significantly increased. By migrating highly correlated data tables in advance, problems such as low read efficiency or access efficiency caused by their absence from the first storage medium can be avoided.

[0082] for example, The data tables in the first data table set, For the data tables in the second data table set, the electronic device calculates all data tables in the first data table set and all data tables in the second data table set within a second preset time period. The pairwise correlation, that is, the electronic device obtains the data table tuple. The electronic device then selects several data table tuples with a relevance higher than the fourth preset threshold, assuming... For the selected data table tuple, the data table For the target data table, the data table To the data table The primary storage medium was migrated.

[0083] In some embodiments, calculating the correlation between data tables in the first data table set and data tables in the second data table set includes: Get the total number of times each data table in the first data table set is accessed within the second preset time period, the total number of times each data table in the second data table set is accessed within the second preset time period, and the number of times the first data table and the second data table are accessed simultaneously within the second preset time period; the first data table is a data table in the first data table set, and the second data table is a data table in the second data table set; The relevance is obtained based on the total number of times each data table in the first data table set is accessed, the total number of times each data table in the second data table set is accessed, and the number of times the first data table and the second data table are accessed simultaneously.

[0084] In this embodiment, the relevance calculation formula can be: . This represents the total number of times the first data table in the first data table set was accessed. This represents the total number of times the second data table in the second data table set was accessed. The correlation is the number of times the first and second data tables are accessed simultaneously. The correlation value ranges from [0, 1]. The higher the value, the greater the probability that the first and second data tables are accessed simultaneously, and the higher the correlation between the first and second data tables.

[0085] Understandably, in this embodiment, the electronic device considers the relevance and affinity of accessed data tables during the cross-domain migration and allocation process. The electronic device proposes a method for calculating the relevance of data tables during joint access and adjusts the allocation results accordingly, ensuring that data tables with high relevance are allocated to the same storage medium.

[0086] As can be seen from the above, the electronic device in this embodiment reduces the access latency of upper-layer application load by considering data relevance during the storage medium adjustment phase. Furthermore, to address the data relevance issue in tiered storage system scenarios, during cross-domain data migration in this embodiment, the electronic device comprehensively considers factors such as the future access probability of the data table, the size of the corresponding table, and the data table relevance, and periodically migrates the data tables. On one hand, the electronic device determines the storage medium of the data table based on the future access probability and the size of the corresponding table; on the other hand, storing data tables with high relevance in the same storage medium effectively improves the response time for joint access by upper-layer application load.

[0087] Understandably, in steps 101 to 104 above, the electronic device can train the first model using access data from at least one data table accessed within a first preset time period to obtain a second model. The electronic device then uses the second model to predict the access status of each data table within the at least one data table and obtains the predicted access probability for each data table output by the second model. The electronic device then determines the corresponding storage medium for each data table based on the predicted access probability. In this process, the second model in the electronic device is obtained by training the first model using access data from at least one data table accessed within the first preset time period. Therefore, the second model obtained by the electronic device can be effectively trained and has high accuracy. Consequently, the accuracy of the predicted access probability for each data table output by the second model is also high. The electronic device can also more accurately determine the corresponding storage medium for each data table based on the predicted access probability. This allows the electronic device to predict the access status of data tables in advance and accurately, enabling the electronic device to adjust the storage medium of the data tables in a timely manner, ensuring that the data tables are stored in suitable storage media and avoiding waste of storage resources.

[0088] Understandably, with the rapid development of cloud computing, big data, and the Internet of Things, enterprises need to process and store massive amounts of data, leading to an increasing demand for efficient data access. Furthermore, user expectations are constantly rising, with higher requirements for the responsiveness and stability of applications and services. To cope with diverse and dynamically changing application loads, migrating services and data to the cloud has become a trend, and various cloud service providers in the market are continuously optimizing their services. Intelligent allocation of elastic caching and elastic storage has become an important component of cloud computing infrastructure. However, current elastic storage and elastic caching on the market cannot accurately detect changes in data access patterns of upper-layer applications, leading to problems such as low query efficiency and low storage resource utilization.

[0089] The embodiments of this application, by predicting data access in real time and dynamically adjusting the storage medium for caching or storage, can help intelligent elastic caching or storage solutions be more widely used in hybrid cloud and multi-cloud environments. This provides more flexible and efficient caching or storage services, meeting users' service quality requirements at a lower cost, improving the performance and resource utilization of upper-layer applications, and thus reducing operating costs. The embodiments of this application have broad application prospects and market demand in fields such as big data, cloud computing, and the Internet of Things.

[0090] The following situations may indicate that other electronic devices have used embodiments of this application: They can be identified by reviewing publicly available information such as white papers and official websites of other electronic devices, and examining the technical descriptions and usage methods therein to determine whether they use the data access prediction features of embodiments of this application. Alternatively, they can be identified by reviewing product manuals, product posters, user manuals, and other product sales materials related to the solutions of other electronic devices, to determine the specific steps and technical details of their operation, such as the generation method of real-time data streams, the rules for generating training data, the implementation method of online training, and the adjustment strategy of storage media, thereby determining whether other electronic devices have used embodiments of this application. Alternatively, they can be identified by analyzing the source code of other electronic devices to determine whether they possess the same methods, parameters, or architectural style as those in this application.

[0091] See Figure 6 , Figure 6 This is a schematic diagram of the structure of a data table storage device provided in an embodiment of this application, such as... Figure 6 As shown, the data table storage device 600 includes: The acquisition module 601 is used to acquire access data accessed in at least one data table within a first preset time period; Training module 602 is used to train the first model based on the access data accessed in the at least one data table to obtain the second model; The prediction module 603 is used to predict the access status of each data table through the second model and obtain the predicted access probability of each data table output by the second model. The first determining module 604 is used to determine the storage medium corresponding to each data table based on the predicted access probability corresponding to each data table.

[0092] Optionally, the access data includes at least one of the following: the table size corresponding to the data table, the number of rows occupied by the data table, the number of columns occupied by the data table, the data table creation time interval, at least one access time interval, and whether the data table was accessed within the first preset time period, wherein the data table creation time interval is the difference between the training time point corresponding to the first model and the data table creation time point, and the access time interval is the difference between two adjacent access timestamps of the data table.

[0093] Optionally, the training module 602 may also include: The first training submodule is used to train the first model based on the access data accessed in the at least one data table, and to obtain the predicted access status of each data table in the at least one data table output by the first model within a preset time window. The comparison submodule is used to compare the actual access status and the predicted access status of each data table in the at least one data table within the preset time window to obtain the accuracy of the first model. The second training submodule is used to continue executing the step of training the first model based on the access data accessed in the at least one data table when the accuracy of the first model is less than or equal to the first preset threshold, until the accuracy is greater than the first preset threshold, so as to obtain the second model.

[0094] Optionally, the first determining module 604 may also include: The first determining submodule is used to determine the storage score of each data table based on the predicted access probability of each data table and the table size of each data table. The first storage submodule is used to store data tables whose storage scores are greater than the second preset threshold to the first storage medium, and to store data tables whose storage scores are less than or equal to the second preset threshold to the second storage medium.

[0095] Optionally, the first determining submodule may also include: The normalization subunit is used to obtain the normalized table size of each data table based on the sum of the table sizes of all data tables and the table size of each data table. The weighted subunit is used to perform a weighted summation of the table size corresponding to each normalized data table and the predicted access probability corresponding to each data table to obtain the storage score corresponding to each data table.

[0096] Optionally, the first storage medium includes a memory pool and a cloud disk, and the second storage medium includes a hard disk; the first storage submodule may further include: The first storage subunit is used to store data tables whose storage scores are greater than the second preset threshold but less than or equal to the third preset threshold to the cloud disk; the third preset threshold is greater than the second preset threshold. The second storage subunit is used to store data tables whose storage scores are greater than the third preset threshold into the memory pool; The third storage subunit is used to store data tables whose storage scores are less than or equal to the second preset threshold to the hard disk.

[0097] Optionally, the data table storage device 600 may also include: The calculation module is used to calculate the correlation between data tables in the first data table set and data tables in the second data table set; the data tables in the first data table set are stored in the first storage medium, and the data tables in the second data table set are stored in the second storage medium. The second determining module is used to determine the data tables in the second data table set whose relevance is greater than the fourth preset threshold as target data tables; A storage module is used to store the target data table to the first storage medium.

[0098] Optionally, the computing module may also include: The acquisition submodule is used to acquire the total number of times each data table in the first data table set is accessed within the second preset time period, the total number of times each data table in the second data table set is accessed within the second preset time period, and the number of times the first data table and the second data table are accessed simultaneously within the second preset time period; the first data table is a data table in the first data table set, and the second data table is a data table in the second data table set; The calculation submodule is used to obtain the relevance based on the total number of times each data table in the first data table set is accessed, the total number of times each data table in the second data table set is accessed, and the number of times the first data table and the second data table are accessed simultaneously.

[0099] The data table storage device 600 is designed to implement the various processes described above in the embodiments of the data table storage method. The technical features are one-to-one and can achieve the same technical effect. To avoid repetition, they will not be described again here.

[0100] This application also provides an electronic device. Since the principle by which this electronic device solves the problem is similar to the data table storage method in the embodiments of this application, the implementation of this electronic device can be found elsewhere. Figure 1 The implementation of the method shown will not be repeated here. Figure 7 As shown, the electronic device of this application embodiment includes: a memory 720 and a processor 710. The processor 710 is used to read the program in the memory 720 and execute the following processes: Retrieve access data from at least one data table within a first preset time period; The first model is trained based on the access data accessed in at least one of the data tables to obtain the second model; The second model is used to predict the access status of each data table, and the predicted access probability of each data table is obtained from the output of the second model. Based on the predicted access probability of each data table, the storage medium corresponding to each data table is determined.

[0101] Optionally, the access data includes at least one of the following: the table size corresponding to the data table, the number of rows occupied by the data table, the number of columns occupied by the data table, the data table creation time interval, at least one access time interval, and whether the data table was accessed within the first preset time period, wherein the data table creation time interval is the difference between the training time point corresponding to the first model and the data table creation time point, and the access time interval is the difference between two adjacent access timestamps of the data table.

[0102] Optionally, the processor 710 is also used to read the program from the memory 720 and perform the following steps: The step of training the first model based on access data accessed in at least one of the data tables to obtain the second model includes: The first model is trained based on the access data accessed in the at least one data table, and the predicted access status of each data table in the at least one data table output by the first model within a preset time window is obtained. The accuracy of the first model is obtained by comparing the actual access status of each data table in the at least one data table within the preset time window with the predicted access status. If the accuracy of the first model is less than or equal to the first preset threshold, the step of training the first model based on the access data accessed in the at least one data table continues until the accuracy is greater than the first preset threshold, so as to obtain the second model.

[0103] Optionally, the processor 710 is also used to read the program from the memory 720 and perform the following steps: The step of determining the storage medium corresponding to each data table based on the predicted access probability of each data table includes: Based on the predicted access probability of each data table and the table size of each data table, the storage score of each data table is determined. Data tables whose storage scores are greater than the second preset threshold are stored in the first storage medium, and data tables whose storage scores are less than or equal to the second preset threshold are stored in the second storage medium.

[0104] Optionally, the processor 710 is also used to read the program from the memory 720 and perform the following steps: The process of determining the storage score for each data table based on the predicted access probability and the table size of each data table includes: Based on the sum of the table sizes of all data tables and the table size corresponding to each data table, the normalized table size corresponding to each data table is obtained. The storage score for each data table is obtained by weighted summing of the table size corresponding to each normalized data table and the predicted access probability corresponding to each data table.

[0105] Optionally, the first storage medium includes a memory pool and a cloud disk, and the second storage medium includes a hard disk; the processor 710 is further configured to read the program in the memory 720 and perform the following steps: The step of storing data tables with storage scores greater than a second preset threshold to a first storage medium, and storing data tables with storage scores less than or equal to the second preset threshold to a second storage medium, includes: Data tables whose storage scores are greater than the second preset threshold but less than or equal to the third preset threshold are stored in the cloud disk; the third preset threshold is greater than the second preset threshold. Store the data tables whose storage scores are greater than the third preset threshold into the memory pool; Data tables whose storage scores are less than or equal to the second preset threshold are stored on the hard disk.

[0106] Optionally, the processor 710 is also used to read the program from the memory 720 and perform the following steps: Calculate the correlation between the data tables in the first data table set and the data tables in the second data table set; the data tables in the first data table set are stored in the first storage medium, and the data tables in the second data table set are stored in the second storage medium; The data tables in the second set of data tables whose relevance is greater than the fourth preset threshold are determined as target data tables; The target data table is stored in the first storage medium.

[0107] Optionally, the processor 710 is also used to read the program from the memory 720 and perform the following steps: The calculation of the correlation between the data tables in the first data table set and the data tables in the second data table set includes: Get the total number of times each data table in the first data table set is accessed within the second preset time period, the total number of times each data table in the second data table set is accessed within the second preset time period, and the number of times the first data table and the second data table are accessed simultaneously within the second preset time period; the first data table is a data table in the first data table set, and the second data table is a data table in the second data table set; The relevance is obtained based on the total number of times each data table in the first data table set is accessed, the total number of times each data table in the second data table set is accessed, and the number of times the first data table and the second data table are accessed simultaneously.

[0108] Among them, Figure 7 In this context, the bus architecture can include any number of interconnected buses and bridges, specifically linking various circuits of one or more processors represented by processor 710 and memory represented by memory 720 together. The bus architecture can also link various other circuits such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and therefore will not be described further herein. The bus interface provides the interface.

[0109] This application also provides a computer-readable storage medium storing a computer program. When executed by a processor, the computer program implements the various processes of the above-described data table storage method embodiments and achieves the same technical effects. To avoid repetition, it will not be described again here. The computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, etc.

[0110] This application also provides a computer program product, including computer instructions. When executed by a processor, these computer instructions implement the various processes of the above-described data table storage method embodiments and achieve the same technical effects. To avoid repetition, they will not be described again here.

[0111] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0112] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0113] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. A data table storage method, characterized in that, The method includes: Retrieve access data from at least one data table within a first preset time period; The first model is trained based on the access data accessed in at least one of the data tables to obtain the second model; The second model is used to predict the access status of each data table, and the predicted access probability of each data table is obtained from the output of the second model. Based on the predicted access probability of each data table, the storage medium corresponding to each data table is determined.

2. The method according to claim 1, characterized in that, The access data includes at least one of the following: the table size corresponding to the data table, the number of rows in the data table, the number of columns in the data table, the data table creation time interval, at least one access time interval, and whether the data table was accessed within the first preset time period. The data table creation time interval is the difference between the training time point corresponding to the first model and the data table creation time point, and the access time interval is the difference between two adjacent access timestamps of the data table.

3. The method according to claim 1, characterized in that, The step of training the first model based on access data accessed in at least one of the data tables to obtain the second model includes: The first model is trained based on the access data accessed in the at least one data table, and the predicted access status of each data table in the at least one data table output by the first model within a preset time window is obtained. The accuracy of the first model is obtained by comparing the actual access status of each data table in the at least one data table within the preset time window with the predicted access status. If the accuracy of the first model is less than or equal to the first preset threshold, the step of training the first model based on the access data accessed in the at least one data table continues until the accuracy is greater than the first preset threshold, so as to obtain the second model.

4. The method according to claim 1, characterized in that, The step of determining the storage medium corresponding to each data table based on the predicted access probability of each data table includes: Based on the predicted access probability of each data table and the table size of each data table, the storage score of each data table is determined. Data tables whose storage scores are greater than the second preset threshold are stored in the first storage medium, and data tables whose storage scores are less than or equal to the second preset threshold are stored in the second storage medium.

5. The method according to claim 4, characterized in that, The process of determining the storage score for each data table based on the predicted access probability and the table size of each data table includes: Based on the sum of the table sizes of all data tables and the table size corresponding to each data table, the normalized table size corresponding to each data table is obtained. The storage score for each data table is obtained by weighted summing of the table size corresponding to each normalized data table and the predicted access probability corresponding to each data table.

6. The method according to claim 4, characterized in that, The first storage medium includes a memory pool and a cloud disk, and the second storage medium includes a hard disk; the step of storing data tables with storage scores greater than a second preset threshold to the first storage medium, and storing data tables with storage scores less than or equal to the second preset threshold to the second storage medium, includes: Data tables whose storage scores are greater than the second preset threshold but less than or equal to the third preset threshold are stored in the cloud disk; the third preset threshold is greater than the second preset threshold. Store the data tables whose storage scores are greater than the third preset threshold into the memory pool; Data tables whose storage scores are less than or equal to the second preset threshold are stored on the hard disk.

7. The method according to claim 1, characterized in that, The method further includes: Calculate the correlation between the data tables in the first data table set and the data tables in the second data table set; the data tables in the first data table set are stored in the first storage medium, and the data tables in the second data table set are stored in the second storage medium; The data tables in the second set of data tables whose relevance is greater than the fourth preset threshold are determined as target data tables; The target data table is stored in the first storage medium.

8. The method according to claim 7, characterized in that, The calculation of the correlation between the data tables in the first data table set and the data tables in the second data table set includes: Get the total number of times each data table in the first data table set is accessed within the second preset time period, the total number of times each data table in the second data table set is accessed within the second preset time period, and the number of times the first data table and the second data table are accessed simultaneously within the second preset time period; the first data table is a data table in the first data table set, and the second data table is a data table in the second data table set; The relevance is obtained based on the total number of times each data table in the first data table set is accessed, the total number of times each data table in the second data table set is accessed, and the number of times the first data table and the second data table are accessed simultaneously.

9. A data table storage device, characterized in that, include: The acquisition module is used to acquire access data from at least one data table within a first preset time period. The training module is used to train the first model based on the access data accessed in the at least one data table to obtain the second model; The prediction module is used to predict the access status of each data table through the second model and obtain the predicted access probability of each data table output by the second model. The first determining module is used to determine the storage medium corresponding to each data table based on the predicted access probability of each data table.

10. An electronic device, characterized in that, include: A processor, a memory, and a program stored in the memory and executable on the processor, wherein the program, when executed by the processor, implements the steps of the data table storage method as described in any one of claims 1 to 8.

11. A computer-readable storage medium for storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps in the data table storage method as described in any one of claims 1 to 8.

12. A computer program product, characterized in that, It includes computer instructions that, when executed by a processor, implement the steps in the data table storage method as described in any one of claims 1 to 8.

Images