Database task processing method, device and storage medium

Abstract

The embodiment of the present application provides a database task processing method, device and storage medium. In the embodiment of the present application, a machine learning model is integrated in the database storage engine, and the data samples and resources used in the machine learning model training are dynamically adjusted by sensing the load pressure of the database storage engine. Through data collection at the storage engine level and the design of a lightweight model, the model is updated online based on background tasks, thereby ensuring the effectiveness and availability of the machine learning model integrated in the database storage engine, and improving the accuracy and recall rate of the task execution results.

Description

Technical Field

[0001] The present application relates to the field of database technology, and in particular to a database task processing method, device and storage medium. Background Art

[0002] As a complex system, the database has many parameters from the computing layer to the storage layer. Behind these parameters correspond many tasks triggered by rules. Although the method of triggering tasks based on fixed rules is simple to implement, the accuracy and recall rate of the task results are low.

[0003] For example, some databases use a Log Structured Merge-Tree (LSM-Tree) storage structure. LSM-Tree uses a rule-based hot and cold identification algorithm, such as Least Recently Used (LRU) or Least Frequently Used (LFU) to identify the hotness or coldness of data, and stores data in layers according to the hotness or coldness of the data. Although this can reduce storage costs and maintain high database performance, the recognition accuracy and recall rate of rule-based hot and cold data identification are low. Summary of the invention

[0004] Multiple aspects of the present application provide a database task processing method, device and storage medium to improve the accuracy and recall rate of task execution results.

[0005] An embodiment of the present application provides a database task processing method, which is applicable to a database storage engine, wherein the database storage engine is integrated with a machine learning model for processing database tasks, and the method comprises: when the database task is triggered, calling the machine learning model to execute the database task, and triggering subsequent actions according to the task execution result output by the machine learning model; and when the database task is not triggered, if the set model dynamic update condition is met, determining the target resource information and target sample data available for this model update according to the current load information of the database storage engine, and starting a background task to update the machine learning model online according to the target resource information and target sample data.

[0006] An embodiment of the present application also provides a hot and cold data processing method, which is applicable to a database storage engine, wherein the database storage engine is integrated with a machine learning model for identifying hot and cold data in a data tiered application scenario, and the database storage engine includes K layers of persistent storage media in the data tiered application scenario. The method includes: monitoring the usage of any layer of persistent storage media in the first M layers of persistent storage media; when the usage rate of the persistent storage media reaches a set usage rate threshold, calling the machine learning model to identify hot and cold data for data blocks in the persistent storage media, and outputting the cold and hot data identification results; according to the cold and hot data identification results, compressing the cold data in the persistent storage medium and merging them into the next layer of persistent storage media, and pre-storing the hot data in the persistent storage medium into the memory of the database storage engine; K is a positive integer greater than or equal to 2, and M is a positive integer less than K.

[0007] An embodiment of the present application also provides a database storage engine, in which a machine learning model for processing database tasks is integrated, and the data storage engine includes: a model calling module, an action triggering module and a model updating module; the model calling module is used to call the machine learning model to execute the database task when the action triggering module triggers the database task, and trigger subsequent actions according to the task execution result output by the machine learning model; the model updating module is used to determine the target resource information and target sample data available for this model update according to the current load information of the database storage engine if the set model dynamic update conditions are met when the action triggering module does not trigger the database task, and start a background task to update the machine learning model online according to the target resource information and target sample data.

[0008] An embodiment of the present application also provides a data processing device, which runs a database storage engine, and the database storage engine is integrated with a machine learning model for processing database tasks. The data processing device includes: a memory and a processor; the memory is used to store a computer program corresponding to the database storage engine; the processor is used to execute the computer program corresponding to the database storage engine, so as to: when the database task is triggered, call the machine learning model to execute the database task, and trigger subsequent actions according to the task execution result output by the machine learning model; and when the database task is not triggered, if the set model dynamic update condition is met, determine the target resource information and target sample data available for this model update according to the current load information of the database storage engine, and start a background task to update the machine learning model online according to the target resource information and target sample data.

[0009] The embodiment of the present application also provides a hot and cold data processing device, which runs a database storage engine, and the database storage engine is integrated with a machine learning model for identifying hot and cold data in a data tiered application scenario. The data processing device includes: a first memory and a processor; the first memory is used to store a computer program corresponding to the database storage engine; in the data tiered application scenario, the data processing device also includes: a second memory, the second memory is composed of K layers of persistent storage media; the processor is used to execute the computer program corresponding to the database storage engine, so as to: monitor the usage of any layer of persistent storage media in the first M layers of persistent storage media; when the usage rate of the persistent storage medium reaches a set usage rate threshold, call the machine learning model to identify hot and cold data for the data blocks in the persistent storage medium, and output the cold and hot data identification results; according to the cold and hot data identification results, compress the cold data in the persistent storage medium and merge it into the next layer of persistent storage medium, and pre-store the hot data in the persistent storage medium into the memory of the database storage engine; K is a positive integer greater than or equal to 2, and M is a positive integer less than K.

[0010] The embodiment of the present application also provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the processor implements the steps in the database task processing method or the hot and cold data processing method provided in the embodiment of the present application.

[0011] An embodiment of the present application also provides a computer program product, including a computer program / instruction. When the computer program / instruction is executed by a processor, the processor is caused to implement the steps in the database task processing method or the hot and cold data processing method provided in the embodiment of the present application.

[0012] In an embodiment of the present application, a machine learning model is integrated in a database storage engine, and the data samples and resources used in the machine learning model training are dynamically adjusted by sensing the load pressure of the database storage engine. Through data collection at the storage engine level and the design of a lightweight model, the model is updated online based on background tasks, thereby ensuring the effectiveness and availability of the machine learning model integrated in the database storage engine and improving the accuracy and recall of task execution results. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0014] Figure 1 A flowchart of a database task processing method provided for an exemplary embodiment of the present application;

[0015] Figure 2 A flowchart of a method for processing hot and cold data provided by an exemplary embodiment of the present application;

[0016] Figure 3a A hierarchical schematic diagram of a database storage engine provided for an exemplary embodiment of the present application;

[0017] Figure 3b This is a comparison effect diagram of the technical solution of the exemplary embodiment of the present application and two traditional solutions, LRU and LFU, in terms of delay performance;

[0018] Figure 4 A schematic diagram of the structure of a database storage engine provided for an exemplary embodiment of the present application;

[0019] Figure 5 A structural schematic diagram of a data processing device provided for an exemplary embodiment of the present application;

[0020] Figure 6 A schematic structural diagram of a cold and hot data processing device provided for an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0021] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application will be clearly and completely described below in combination with the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.

[0022] In the prior art, although the method of triggering tasks based on fixed rules is simple to implement, it does not use the historical information in the database and often cannot provide sufficient accuracy. In addition, it is difficult to make adaptive adjustments to fixed rules for different loads, resulting in low accuracy and recall of database task execution results. In an embodiment of the present application, a machine learning model is integrated in the database storage engine, and the data samples and resources used in the machine learning model training are dynamically adjusted by sensing the load pressure of the database storage engine. Through data collection at the storage engine level and the design of a lightweight model, the model is updated online based on background tasks, thereby ensuring the effectiveness and availability of the machine learning model integrated in the database storage engine and improving the accuracy and recall of task execution results.

[0023] The technical solutions provided by various embodiments of the present application are described in detail below in conjunction with the accompanying drawings.

[0024] Figure 1 A flowchart of a database task processing method provided for an exemplary embodiment of the present application is shown as follows: Figure 1 As shown, the database task processing method is applicable to a database storage engine, and the database storage engine is integrated with a machine learning model for processing database tasks. The database task processing method includes:

[0025] 101. When a database task is triggered, the machine learning model is called to execute the database task, and subsequent actions are triggered according to the task execution results output by the machine learning model;

[0026] 102. When the database task is not triggered, if the set model dynamic update conditions are met, the target resource information and target sample data available for this model update are determined based on the current load information of the database storage engine, and based on the target resource information and target sample data, the background task is started to update the machine learning model online.

[0027] In this embodiment, the database storage engine is a software organization at the bottom of the database. The database storage engine can implement a variety of database tasks, such as storage restrictions, transaction security, locking mechanisms, indexes, data caches, index caches, etc. In this embodiment, the machine learning model is integrated into the database storage engine, and the database storage engine can process database tasks based on the machine learning model, such as cold and hot data identification, task scheduling, index updates, learned indexes (Learned Index) or learned optimizers (Learned Optimizer) in data stratification scenarios. Among them, the machine learning model can be but is not limited to: Logistic Regression, decision tree, support vector machine, random forest, naive Bayes, k-nearest neighbor (kNN) algorithm or deep learning model, etc.

[0028] In this embodiment, the triggering method of the database task is not limited. In an optional embodiment, the database task is periodically triggered by a background timer task. In another optional embodiment, the database task is manually triggered by the maintenance personnel of the database storage engine. In yet another optional embodiment, the database task is triggered by a specific event. Depending on the application scenario, the specific event that triggers the database task is also different. For example, in a data stratification application scenario, the database task is a cold and hot data identification task. The specific event that triggers the cold and hot data identification is that the usage rate of the storage medium storing temporary data reaches a set usage threshold, which triggers the cold and hot identification task for the temporary data. For another example, in an index update scenario, the database task is an index update task. The specific event that triggers the index update task is that the amount of data added, deleted, modified or queried in the database reaches a set threshold, which triggers the index update task.

[0029] In this embodiment, when a database task is triggered, a machine learning model can be called to execute the database task, and a subsequent action can be triggered according to the task execution result output by the machine learning model. According to different application scenarios, the task execution result output by the machine learning model is different, and accordingly, the subsequent actions triggered are also different. For example, in the process of identifying hot and cold data in a data stratification application scenario, the task execution result output by the machine learning model is which data blocks are hot data and which data blocks are hot data. The subsequent action is to store the hot data in a storage medium with a faster read and write speed or an expensive price, such as a solid state drive (Solid State Disk or Solid State Drive, SSD), and store the cold data in a storage medium with a slower read and write speed or a lower price, such as a hard disk drive (Hard Disk Drive, HDD).

[0030] In this embodiment, when the database task is not triggered, if the set model dynamic update condition is met, the machine learning model is dynamically updated. The set model dynamic update condition can be a background scheduled task that triggers the model update, and the background scheduled task can trigger the model update every 24 hours, every week, or every month. In addition, the dynamic update of the model can also be manually triggered by the maintenance personnel of the database storage engine.

[0031] In this embodiment, when the set model dynamic update conditions are met, the target resource information and target sample data available for this model update can be determined based on the current load information of the database storage engine. Among them, the current load information can reflect the current load pressure of the database storage engine. Optionally, the current load information can be reflected by the current resource usage status information of the database storage engine. In other words, the current resource usage status information characterizes the current load information of the database storage engine. The current resource usage status information can be but is not limited to: CPU usage status information, memory usage status information, and hard disk usage status information. Among them, the target resource information represents the resource information that can be occupied by the machine learning model during the dynamic update process, such as CPU, memory, or hard disk resources. The target sample quantity refers to the sample data that can be used in the process of updating the machine learning model. If the load pressure of the database storage engine is small, it means that the current database storage engine occupies small resources. In the process of dynamically updating the machine learning model, more target resource information and target sample data can be allocated to the model update process; on the contrary, if the load pressure of the database storage engine is large, it means that the current database storage engine occupies large resources. In the process of dynamically updating the machine learning model, less target resource information and target sample data can be allocated to the model update process, or the update of this model can be skipped directly. After determining the target resource information and target sample data, the background task can be started to update the machine learning model online according to the target resource information and target sample data. Among them, according to the load pressure of the database storage engine, that is, the current load information, the resources required for the model update and the sample data involved in the model update are dynamically adjusted, which can increase the complexity of the model update as much as possible without affecting the normal processing of transactions by the database storage engine, thereby ensuring the effectiveness and availability of the machine learning model integrated in the database storage engine and improving the accuracy and recall rate of the task results. In addition, in the present embodiment, a machine learning model that can process database tasks is integrated into the database storage engine. Compared with the method of using the machine learning model as an external service, the operation and maintenance cost and control complexity of the database storage engine for external services are reduced; in addition, in the embodiment of the present application, there is no need to transmit the target sample data to the external service during the update of the machine learning model, thereby saving transmission costs.

[0032] In an optional embodiment, a method for determining the target resource information and target sample data available for this model update based on the current load information of the database storage engine includes: obtaining the current resource usage status information of the database storage engine; determining the target resource information and training parameters available for this model update based on the current resource usage status information, the training parameters including at least the number of samples; and sampling the target sample data from the kernel state data corresponding to the database task based on the number of samples.

[0033] Among them, training parameters are parameters required in the process of updating the machine learning model. For example, training parameters include but are not limited to: model hyperparameters, number of samples or number of iterations, etc. Model hyperparameters are configuration parameters external to the model. The values ​​of hyperparameters can be configured manually or use default parameter values. Model hyperparameters can be but are not limited to: model learning rate, number of training threads, number of hidden layers of deep neural networks or depth of trees, etc. The kernel state data corresponding to the database task is data generated by a database storage engine working in kernel state to complete the database task. For example, kernel state data may include but is not limited to: historical access information or static properties of data blocks involved in the database engine processing database tasks, throughput, response time and other indicator information involved in the database engine processing database tasks. Static properties of data blocks may include but are not limited to the size of the data in the data block or the tablespace to which the data belongs, etc.

[0034] In this embodiment, the database storage engine can pre-train the machine learning model to obtain the corresponding relationship between the amount of resources for training the machine learning model and the training parameters. If the amount of resources for training the machine learning model is small, the training parameters corresponding to the smaller amount of resources are determined. For example, the training parameters may be a smaller number of training threads, a smaller number of iterations, a smaller number of training samples, or a larger learning rate, etc. If the amount of resources for training the machine learning model is large, the training parameters corresponding to the larger amount of resources are determined. For example, the training parameters may be a larger number of training threads, a larger number of iterations, a larger number of training samples, or a smaller learning rate, etc. Based on this, in an optional embodiment of the present application, the target resource information available for this model update can be determined based on the current resource usage status information; based on the target resource information, combined with the correspondence between the amount of resources obtained in advance and the training parameters, the training parameters available for this model update are determined. For example, if the current resource usage status information indicates that the load pressure of the current database storage engine is relatively small, more target resource information can be allocated for this model update, and based on the target resource information, the available training parameters for this model update can be determined; if the current resource usage status information indicates that the load pressure of the current database storage engine is relatively large, less target resource information can be allocated for this model update, or even the model update can be skipped.

[0035] In an optional embodiment, when the training parameters of the machine learning model include: the number of samples, the number of threads, the number of iterations and model hyperparameters, an implementation method of starting a background task to perform online update of the machine learning model based on target resource information and target sample data includes: starting the thread used for this model update based on the target resource information and the number of threads, and controlling the thread to perform online update of the machine learning model based on the target sample data, the number of iterations and model hyperparameters.

[0036] In some embodiments of the present application, the database task can be implemented as a cold and hot data identification task in a data tiering application scenario. In this application scenario, the present application also provides a cold and hot data processing method, which is applicable to a database storage engine. The database storage engine is integrated with a machine learning model for identifying cold and hot data in a data tiering application scenario, such as Figure 2 As shown, the method includes:

[0037] 201. For any layer of persistent storage media in the first M layers of persistent storage media, monitor the usage of the persistent storage media;

[0038] 202. When the usage rate of the persistent storage medium reaches a set usage rate threshold, call the machine learning model to identify hot and cold data for the data blocks in the persistent storage medium, and output the cold and hot data identification result;

[0039] 203. According to the cold and hot data identification results, the cold data in the persistent storage medium is compressed and merged into the next layer of persistent storage medium, and the hot data in the persistent storage medium is pre-stored in the memory of the database storage engine; K is a positive integer greater than or equal to 2, and M is a positive integer less than K.

[0040] In this embodiment, the database storage engine includes: K layers of persistent storage media, K is a positive integer greater than or equal to 2. Among them, the first M layers are used to persistently store data from the memory, and M is a positive integer less than K. The data in the first M layers of persistent storage media can be cold data or hot data. Cold data refers to data with low access frequency and low efficiency requirements, and hot data refers to data with high access frequency and high processing efficiency requirements. Figure 3a In the figure, the LSM-tree-based database storage engine includes three layers of persistent storage media as an example, but it is not limited to this. The three layers of persistent storage media are: L0 layer persistent storage medium, L1 layer persistent storage medium and L2 layer persistent storage medium. Among them, the L0 and L1 layer persistent storage media are used to persistently store data from the memory, and the data in the L0 and L1 layer persistent storage media can be cold data or hot data. The data in the L2 layer persistent storage medium is cold data. Among them, no matter which layer of persistent storage medium, in the data tiering application scenario, the data in the persistent storage medium is stored in data blocks, and the size of the data block can be 64KB, 512KB or 64MB, etc.

[0041] In this embodiment, the usage of the K-layer persistent storage medium is constantly changing. Specifically, the database storage engine can receive a user's access request (Request), and perform add, delete, modify, query, read, or write operations on the data blocks in the persistent storage medium according to the user's access request, so that the usage of the persistent storage medium changes. Figure 3a In , the user's access request is represented by "Req". Among them, the hot data of the persistent storage medium can be stored in the memory, for example Figure 3a The data block cache (Block Cache) in the database is stored in the database to facilitate users to quickly read the data in the data block. When writing data, the user can write to the memory table (Memtable). When the memory table exceeds a certain size, it will be frozen in the memory and become an immutable memory table. At the same time, in order not to block the write operation, a new memory table needs to be generated to continue to provide write services. Then the immutable memory table can be persistently stored, that is, the immutable memory table is flushed to the persistent storage medium in the form of data blocks. In addition, the database storage engine can also feedback the processing results of the access request to the user.

[0042] In this embodiment, for any layer of persistent storage media in the first M layers of persistent storage media, the usage of the persistent storage media can be monitored; when the usage rate of the persistent storage media reaches the set usage rate threshold, the machine learning model is called to identify the hot and cold data of the data blocks in the persistent storage media, and the cold and hot data identification results are output, wherein the usage rate threshold can be 75%, 80% or 90%, etc.; according to the cold and hot data identification results, the cold data in the persistent storage media is compressed and merged into the next layer of persistent storage media, and the hot data in the persistent storage media is pre-stored in the memory of the database storage engine; K is a positive integer greater than or equal to 2, and M is a positive integer less than K. Among them, the compression and merging methods may include but are not limited to: temp-aware compaction, high-ratio compression, or dictionary compression, etc.

[0043] Below Figure 3a The database storage engine shown in the figure is used as an example. For the L0 layer and L1 layer persistent storage media, its usage is detected. When the usage rate of the L0 layer or L1 layer persistent storage media reaches the set usage rate threshold, the machine learning model is called to identify the hot and cold data of the data blocks in the L0 layer or L1 layer persistent storage media, and the hot and cold data identification results are output, such as Figure 3aStep ④ in; if the usage rate of the L0 layer persistent storage medium exceeds the set usage rate threshold, then according to the cold and hot data identification result, the data blocks corresponding to the cold data in the L0 layer persistent storage medium are compressed and merged into the L1 layer persistent storage medium, and the data blocks corresponding to the hot data in the L0 layer persistent storage medium are pre-stored (prefetched) into the memory of the database storage engine; if the usage rate of the L1 layer persistent storage medium exceeds the set usage rate threshold, then according to the cold and hot data identification result, the data blocks corresponding to the cold data in the L1 layer persistent storage medium are compressed and merged into the L2 layer persistent storage medium, and the data blocks corresponding to the hot data in the L1 layer persistent storage medium are pre-stored (prefetched) into the memory of the database storage engine.

[0044] In an optional embodiment, during the period when the usage rate of the first M-layer persistent storage medium does not reach the set usage rate threshold, if the set model dynamic update condition is met, the target resource information and target sample data available for this model update are determined according to the current load information of the database storage engine, and the background task is started to update the machine learning model online according to the target resource information and target sample data. For details, please refer to the above embodiment and will not be repeated here.

[0045] like Figure 3a As shown, the database storage engine includes: a resource monitoring module 31, a model updating module 32 and an action triggering module, wherein the action triggering module is not Figure 3a As shown in Figure 3a As shown, the resource monitoring module (Resource Monitor) can monitor the load pressure of the current database storage engine, that is, record the load information of the database storage engine. During the period when the usage rate of the persistent storage medium of the L0 layer or L1 layer does not reach the set usage rate threshold, if the set model dynamic update condition is met, for example, the model dynamic update condition is to trigger the model update at 1:00 a.m. every Monday, when the time reaches 1:00 a.m. on Monday, the model dynamic update condition is met, then the dynamic update of the model is triggered. During the dynamic update process of the model, the model update module 32 sends a query request to the resource monitoring module 31 to query the current load pressure of the database engine; after receiving the query request, the resource monitoring module 31 returns the current load information of the database storage engine to the model update module 32, such as Figure 3a The model update module 32 determines the target resource information and target sample data available for this model update based on the current load information. Specifically, the number of samples for this model update can be determined based on the target resource information available for this model update, and the kernel state data in the feature table can be sampled based on the number of samples to obtain the target sample data, such as Figure 3aThen, the model updating module 32 starts a background task to update the machine learning model online according to the target resource information and the target sample data.

[0046] In an optional embodiment, the database storage engine can collect access characteristics of data blocks, that is, kernel-mode data in the database storage engine. Optionally, the historical access characteristics of data blocks can be collected from the data block dimension. For example, the historical access intervals are recorded at the data block granularity, and the historical access intervals reflect the number of times the data block is accessed. When collecting historical access characteristics, not all historical access characteristics are collected. Instead, a sliding window is maintained for each data block. For each data block in the first M layers of persistent storage media, the access characteristics of the data block in the sliding window are collected from the first M layers of persistent storage media with the sliding window corresponding to the data block, and the access characteristics of the data block are stored in the feature table. The above process of collecting kernel-mode data corresponds to Figure 3a Step ① in Figure 3a In the example, the kernel state data of the collected data block is stored in a pattern map. For example, the pattern map includes: data block A1->access feature B1, access feature B2, ..., Bi; data block A2->access feature C1, access feature C2, ..., Ci, where i is the length of the sliding window. Based on the above, in this embodiment, the access features of N data blocks can be sampled in the feature table corresponding to the cold and hot data identification task according to the number of samples, where N is a positive integer greater than or equal to 2; the static attributes of the N data blocks are obtained from the metadata of each of the N data blocks, and the access features and static attributes of the N data blocks are used as target sample data.

[0047] Compared with the solution of hot and cold data identification by traditional database storage engines, the use of the database storage engine with an internally integrated machine learning algorithm provided by the embodiment of the present application for hot and cold data identification can produce obvious results. In order to reflect the obvious effect of the database storage engine provided by the embodiment of the present application in the identification of hot and cold data, an experimental verification is carried out: a database for simulation is selected, and the database is an LSM-tree hybrid storage system. In order to save storage resources, hot data is stored using SSD and cold data is stored using HDD on the LSM-tree hybrid storage system. The database uses the database storage engine provided by the embodiment of the present application. First, the access pattern is modeled using the historical information of data access in the database, and a machine learning model for hot and cold data identification is constructed in the database storage engine. Next, the two traditional solutions of LRU and LFU and the technical solution provided by the embodiment of the present application are respectively used ( Figure 3bIn this case, the hot and cold data identification test is performed in the SSD used in the database, and the identified cold data is transferred to the HDD, and the hot data is pre-stored in the system memory. In this hot and cold identification process, the delay performance of LRU, LFU and this case is statistically analyzed, and the following is obtained: Figure 3b The comparison of the delay performance of this solution with the two traditional solutions of LRU and LFU is shown in the figure. Figure 3b It can be seen that the delay of the hot and cold separation solution based on machine learning provided in the embodiment of the present application can be reduced by approximately 70%.

[0048] It should be noted that the execution subject of each step of the method provided in the above embodiment can be the same device, or the method can be executed by different devices. For example, the execution subject of step 101 to step 102 can be device A; for another example, the execution subject of step 101 can be device A, and the execution subject of step 102 can be device B; and so on.

[0049] In addition, in some of the processes described in the above embodiments and the accompanying drawings, multiple operations that appear in a specific order are included, but it should be clearly understood that these operations may not be executed in the order in which they appear in this article or executed in parallel, and the sequence numbers of the operations, such as 101, 102, etc., are only used to distinguish between different operations, and the sequence numbers themselves do not represent any execution order. In addition, these processes may include more or fewer operations, and these operations may be executed in sequence or in parallel. It should be noted that the descriptions of "first", "second", etc. in this article are used to distinguish different messages, devices, modules, etc., do not represent the order of precedence, and do not limit the "first" and "second" to be different types.

[0050] Based on the above database task processing method, the present application also provides a database storage engine, in which a machine learning model for processing database tasks is integrated, such as Figure 4 As shown, the database storage engine includes: a model calling module 41, an action triggering module 42 and a model updating module 43.

[0051] The model calling module 41 is used to call the machine learning model to execute the database task when the action triggering module 42 triggers the database task, and trigger subsequent actions according to the task execution result output by the machine learning model;

[0052] The model update module 43 is used to determine the target resource information and target sample data available for this model update based on the current load information of the database storage engine when the action trigger module 42 does not trigger the database task and if the set model dynamic update conditions are met, and start the background task to update the machine learning model online based on the target resource information and target sample data.

[0053] In an optional embodiment, when the model update module 43 determines the target resource information and target sample data available for this model update based on the current load information of the database storage engine, it is specifically used to: obtain the current resource usage status information of the database storage engine, the current resource usage status information represents the current load information of the database storage engine; determine the target resource information and training parameters available for this model update based on the current resource usage status information, the training parameters include at least the number of samples; based on the number of samples, sample the target sample data from the kernel state data corresponding to the database task.

[0054] Optionally, when the model update module 43 determines the target resource information and training parameters available for this model update based on the current resource usage status information, it is specifically used to: determine the target resource information available for this model update based on the current resource usage status information; determine the training parameters available for this model update based on the target resource information and the correspondence between the resource amount and training parameters obtained in advance through training.

[0055] In an optional embodiment, the training parameters also include the number of threads, the number of iterations and the model hyperparameters; when the model update module 43 starts the background task to update the machine learning model online according to the target resource information and the target sample data, it is specifically used to: start the thread used for this model update according to the target resource information and the number of threads, and control the thread to update the machine learning model online according to the target sample data, the number of iterations and the model hyperparameters.

[0056] In an optional embodiment, the database task is a hot and cold data identification task in a data tiered application scenario, and the database storage engine includes K layers of persistent storage media in the data tiered application scenario. Then, when the model update module 43 samples the target sample data from the kernel state data corresponding to the database task according to the number of samples, it is specifically used to: sample the access characteristics of N data blocks in the feature table corresponding to the hot and cold data identification task according to the number of samples, where the feature table stores the access characteristics of each data block collected from the first M layers of persistent storage media; obtain the static attributes of the N data blocks from their respective metadata, and use the access characteristics and static attributes of the N data blocks as the target sample data; N and K are positive integers greater than or equal to 2, and M is a positive integer less than K.

[0057] In an optional embodiment, the database storage engine also includes: a collection module and a storage module; the collection module is used to collect access characteristics of the data block in the sliding window from the first M layers of persistent storage media for each data block in the first M layers of persistent storage media with a sliding window corresponding to the data block; the storage module is used to store the access characteristics of the data block in a feature table.

[0058] In an optional embodiment, the database storage engine also includes: a monitoring module; a monitoring module for monitoring the usage of persistent storage media for any layer of persistent storage media in the first M layers of persistent storage media; an action triggering module for triggering a cold and hot data identification task when the usage rate of the persistent storage media reaches a set usage rate threshold; accordingly, when the database task is triggered, the model update module 43 calls the machine learning model to execute the database task, and is specifically used to: call the machine learning model to perform cold and hot identification on the data blocks in the persistent storage medium, and output the cold and hot identification results; accordingly, when the model calling module 41 triggers subsequent actions according to the task execution results output by the machine learning model, it is specifically used to: according to the cold and hot data identification results, compress the cold data in the persistent storage medium and merge it into the next layer of persistent storage medium, and pre-store the hot data in the persistent storage medium in the memory of the database storage engine.

[0059] Figure 5 A schematic diagram of a data processing device provided for an exemplary embodiment of the present application. The data processing device runs a database storage engine, which is integrated with a machine learning model for processing database tasks, such as Figure 5 As shown, the data processing device includes: a memory 54 and a processor 55.

[0060] The memory 54 is used to store computer programs corresponding to the database storage engine, and can be configured to store various other data to support operations on the data processing device. Examples of such data include instructions for any application program or method operating on the data processing device.

[0061] The memory 54 can be implemented by any type of volatile or nonvolatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk.

[0062] The processor 55 is coupled to the memory 54 and is used to execute a computer program corresponding to the database storage engine in the memory 54, so as to: when a database task is triggered, call the machine learning model to execute the database task, and trigger subsequent actions according to the task execution results output by the machine learning model; and when the database task is not triggered, if the set model dynamic update conditions are met, determine the target resource information and target sample data available for this model update according to the current load information of the database storage engine, and start the background task to update the machine learning model online according to the target resource information and target sample data.

[0063] In an optional embodiment, when the processor 55 determines the target resource information and target sample data available for this model update based on the current load information of the database storage engine, it is specifically used to: obtain current resource usage status information of the database storage engine, the current resource usage status information represents the current load information of the database storage engine; determine the target resource information and training parameters available for this model update based on the current resource usage status information, the training parameters include at least the number of samples; and sample the target sample data from the kernel state data corresponding to the database task based on the number of samples.

[0064] In an optional embodiment, when the processor 55 determines the target resource information and training parameters available for this model update based on the current resource usage status information, it is specifically used to: determine the target resource information available for this model update based on the current resource usage status information; determine the training parameters available for this model update based on the target resource information and the correspondence between the resource amount and the training parameters obtained in advance.

[0065] In an optional embodiment, the training parameters also include the number of threads, the number of iterations and the model hyperparameters; when the processor 55 starts the background task to update the machine learning model online according to the target resource information and the target sample data, it is specifically used to: start the thread used for this model update according to the target resource information and the number of threads, and control the thread to update the machine learning model online according to the target sample data, the number of iterations and the model hyperparameters.

[0066] In an optional embodiment, the database task is a hot and cold data identification task in a data tiered application scenario, and the database storage engine includes K layers of persistent storage media in the data tiered application scenario. When the processor 55 samples the target sample data from the kernel state data corresponding to the database task according to the sample quantity, it is specifically used to: sample the access characteristics of N data blocks from the feature table corresponding to the hot and cold data identification task according to the sample quantity, and the feature table stores the access characteristics of each data block collected from the first M layers of persistent storage media; obtain the static attributes of the N data blocks from their respective metadata, and use the access characteristics and static attributes of the N data blocks as the target sample data; N and K are positive integers greater than or equal to 2, and M is a positive integer less than K.

[0067] In an optional embodiment, the processor 55 is also used to: for each data block in the first M layers of persistent storage media, collect access characteristics of the data block in the sliding window from the first M layers of persistent storage media with a sliding window corresponding to the data block, and store the access characteristics of the data block in a feature table.

[0068] In an optional embodiment, for any layer of persistent storage media in the first M layers of persistent storage media, the processor 55 is also used to: monitor the usage of the persistent storage medium, and trigger the cold and hot data identification task when the usage rate of the persistent storage medium reaches a set usage rate threshold; when the database task is triggered, the processor 55 calls the machine learning model to execute the database task, specifically for: calling the machine learning model to perform cold and hot identification on the data blocks in the persistent storage medium, and output the cold and hot identification results; when the processor 55 triggers subsequent actions according to the task execution results output by the machine learning model, specifically for: according to the cold and hot data identification results, compressing the cold data in the persistent storage medium and merging it into the next layer of persistent storage medium, and pre-storing the hot data in the persistent storage medium into the memory of the database storage engine.

[0069] Further, if Figure 5 As shown, the data processing device also includes: a communication component 56, a display 57, a power component 58, an audio component 59 and other components. Figure 5 Only some components are shown schematically, which does not mean that the data processing device only includes Figure 5 It should be noted that Figure 5 The components in the dotted box are optional components, not mandatory components, and the specific components may depend on the product form of the data processing device.

[0070] The data processing device of this embodiment can be implemented as a terminal device such as a desktop computer, a laptop computer, or a smart phone, or a server device such as a conventional server, a cloud server, or a server array. If the data processing device of this embodiment is implemented as a terminal device such as a desktop computer, a laptop computer, or a smart phone, it can include Figure 5 If the data processing device of this embodiment is implemented as a server-side device such as a conventional server, a cloud server or a server array, it may not include Figure 5 Components within the dashed box.

[0071] Accordingly, an embodiment of the present application further provides a computer-readable storage medium storing a computer program, which, when executed by a processor, enables the processor to implement Figure 1 The steps in the method shown may be executed by a data processing device.

[0072] Accordingly, an embodiment of the present application also provides a computer program product, including a computer program / instruction. When the computer program / instruction is executed by a processor, the processor is caused to implement the steps in the database task processing method provided in the embodiment of the present application.

[0073] Figure 6A schematic diagram of a hot and cold data processing device provided for an exemplary embodiment of the present application. The hot and cold data processing device runs a database storage engine, which is integrated with a machine learning model for processing hot and cold data identification in a data stratification application scenario, such as Figure 6 As shown, the cold and hot data processing device includes: a first memory 63, a second memory 64 and a processor 65.

[0074] The first memory 63 is used to store computer programs corresponding to the database storage engine, and can be configured to store various other data to support operations on the hot and cold data processing devices. Examples of such data include instructions for any application or method used to operate on the hot and cold data processing devices.

[0075] The first memory 63 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0076] The second memory 64 is used to store data blocks in the database storage engine; the second memory 64 is composed of K layers of persistent storage media, where K is a positive integer greater than or equal to 2.

[0077] The processor 66 is coupled to the first memory 63 and is used to execute a computer program corresponding to the database storage engine in the first memory 63, so as to: monitor the usage of the persistent storage medium for any layer of the first M layers of persistent storage medium; when the usage rate of the persistent storage medium reaches a set usage rate threshold, call the machine learning model to identify hot and cold data for the data blocks in the persistent storage medium, and output the cold and hot data identification results; according to the cold and hot data identification results, compress the cold data in the persistent storage medium and merge it into the next layer of persistent storage medium, and pre-store the hot data in the persistent storage medium in the memory of the database storage engine; K is a positive integer greater than or equal to 2, and M is a positive integer less than K.

[0078] In an optional embodiment, the processor 65 is also used to: during the period when the usage rate of the persistent storage medium has not reached the set usage rate threshold, if the set model dynamic update conditions are met, determine the target resource information and target sample data available for this model update based on the current load information of the database storage engine, and start a background task to update the machine learning model online based on the target resource information and target sample data.

[0079] In an optional embodiment, when the processor 65 determines the target resource information and target sample data available for this model update based on the current load information of the database storage engine, it is specifically used to: obtain the current resource usage status information of the database storage engine, the current resource usage status information represents the current load information of the database storage engine; determine the target resource information and training parameters available for this model update based on the current resource usage status information, the training parameters at least include the number of samples; based on the number of samples, sample the access characteristics of N data blocks from the feature table, the feature table stores the access characteristics of each data block collected from the first M layers of persistent storage media; respectively obtain the static attributes of the N data blocks from their respective metadata, and use the access characteristics and static attributes of the N data blocks as the target sample data available for this model update; N is a positive integer greater than or equal to 2.

[0080] Further, if Figure 6 As shown, the hot and cold data processing device also includes: a communication component 66, a display 67, a power component 68, an audio component 69 and other components. Figure 6 Only some components are shown schematically, which does not mean that the cold and hot data processing equipment only includes Figure 6 It should be noted that Figure 6 The components in the dotted box are optional components, not mandatory components, and the specific components may depend on the product form of the cold and hot data processing equipment.

[0081] The hot and cold data processing device of this embodiment can be implemented as a terminal device such as a desktop computer, a laptop computer, or a smart phone, or a server device such as a conventional server, a cloud server, or a server array. If the hot and cold data processing device of this embodiment is implemented as a terminal device such as a desktop computer, a laptop computer, or a smart phone, it can include Figure 5 If the hot and cold data processing device of this embodiment is implemented as a server-side device such as a conventional server, a cloud server or a server array, it may not include Figure 5 Components within the dashed box.

[0082] Accordingly, an embodiment of the present application further provides a computer-readable storage medium storing a computer program, which, when executed by a processor, enables the processor to implement Figure 2 The steps in the method shown may be performed by hot and cold data processing equipment.

[0083] Accordingly, an embodiment of the present application also provides a computer program product, including a computer program / instruction. When the computer program / instruction is executed by a processor, the processor is caused to implement the steps in the hot and cold data processing method provided in the embodiment of the present application.

[0084] Above Figure 5 and Figure 6 The communication component in is configured to facilitate wired or wireless communication between the device where the communication component is located and other devices. The device where the communication component is located can access a wireless network based on a communication standard, such as WiFi, 2G, 3G, 4G / LTE, 5G and other mobile communication networks, or a combination thereof. In an exemplary embodiment, the communication component receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0085] Above Figure 5 and Figure 6 The display in the embodiment includes a screen, which may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from a user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundary of a touch or slide action, but also detect the duration and pressure associated with the touch or slide operation.

[0086] Above Figure 5 and Figure 6 The power supply component in the device provides power to various components of the device where the power supply component is located. The power supply component may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the device where the power supply component is located.

[0087] Above Figure 5 and Figure 6 The audio component in can be configured to output and / or input audio signals. For example, the audio component includes a microphone (MIC), and when the device where the audio component is located is in an operating mode, such as a call mode, a recording mode, and a speech recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in a memory or sent via a communication component. In some embodiments, the audio component also includes a speaker for outputting an audio signal.

[0088] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0089] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0090] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0091] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0092] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0093] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0094] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0095] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0096] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. A method for processing hot and cold data, characterized in that: Applicable to a database storage engine, the database storage engine is integrated with a machine learning model for identifying hot and cold data in a data tiering application scenario, the database storage engine includes a K-layer persistent storage medium in the data tiering application scenario, and the method includes: For any layer of persistent storage media in the first M layers of persistent storage media, monitoring the usage of the persistent storage media; When the usage rate of the persistent storage medium reaches a set usage rate threshold, calling the machine learning model to perform cold and hot data identification on the data blocks in the persistent storage medium, and outputting the cold and hot data identification result; According to the cold and hot data identification result, the cold data in the persistent storage medium is compressed and merged into the next layer of persistent storage medium, and the hot data in the persistent storage medium is pre-stored in the memory of the database storage engine; K is a positive integer greater than or equal to 2, and M is a positive integer less than K; During the period when the usage rate of the persistent storage medium has not reached the set usage rate threshold, if the set model dynamic update condition is met, the target resource information and target sample data available for this model update are determined based on the current load information of the database storage engine, and based on the target resource information and target sample data, a background task is started to perform online update of the machine learning model. The current load information reflects the current load pressure of the database storage engine, and the current resource usage status information of the database storage engine represents the current load information of the database storage engine. The target resource information represents information about the resources that can be occupied by the machine learning model during the dynamic update process. The greater the current load pressure, the fewer resources the machine learning model can occupy during the dynamic update process, and the smaller the number of samples corresponding to the target sample data.

2. The method according to claim 1, characterized in that: According to the current load information of the database storage engine, the target resource information and target sample data available for this model update are determined, including: Acquire current resource usage status information of the database storage engine, where the current resource usage status information represents current load information of the database storage engine; Determine target resource information and training parameters available for this model update according to the current resource usage status information, wherein the training parameters include at least the number of samples; According to the number of samples, access features of N data blocks are sampled from a feature table, where the feature table stores access features of each data block collected from the first M layers of persistent storage media; The static properties of the N data blocks are respectively obtained from the metadata of each of the N data blocks, and the access characteristics and static properties of the N data blocks are used as target sample data available for this model update; N is a positive integer greater than or equal to 2.

3. A hot and cold data processing device, running a database storage engine, characterized in that: The database storage engine is integrated with a machine learning model for identifying hot and cold data in a data tiering application scenario. The data processing device includes: a first memory and a processor; the first memory is used to store a computer program corresponding to the database storage engine; in the data tiering application scenario, the data processing device further includes: a second memory, the second memory is composed of a K-layer persistent storage medium; the processor is used to execute the computer program corresponding to the database storage engine, so as to: For any layer of persistent storage media in the first M layers of persistent storage media, monitor the usage of the persistent storage media; when the usage rate of the persistent storage media reaches a set usage rate threshold, call the machine learning model to identify hot and cold data for the data blocks in the persistent storage media, and output the cold and hot data identification results; according to the cold and hot data identification results, compress the cold data in the persistent storage media and merge them into the next layer of persistent storage media, and pre-store the hot data in the persistent storage media into the memory of the database storage engine; K is a positive integer greater than or equal to 2, and M is a positive integer less than K; During the period when the usage rate of the persistent storage medium has not reached the set usage rate threshold, if the set model dynamic update condition is met, the target resource information and target sample data available for this model update are determined according to the current load information of the database storage engine, and based on the target resource information and target sample data, a background task is started to perform online update of the machine learning model; the current load information reflects the current load pressure of the database storage engine, the current resource usage status information of the database storage engine represents the current load information of the database storage engine, and the target resource information represents information about the resources that the machine learning model can occupy during the dynamic update process. The greater the current load pressure, the fewer resources the machine learning model can occupy during the dynamic update process, and the smaller the number of samples corresponding to the target sample data.

4. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the processor is caused to implement the steps in the method according to any one of claims 1 to 2.

Images