Data management method, data management device and storage medium

Abstract

The present invention discloses a data management method, a data management device, and a storage medium. The data management method is applied to a massively parallel processing database (MPPDB). Based on a query request from a user, query behavior information regarding the MPPDB is obtained. Based on the query behavior information, a storage prediction model is generated through machine learning, and a predicted storage state of the MPPDB is obtained based on the storage prediction model. Storage partitions in the MPPDB are adjusted based on the predicted storage state. In an embodiment of the present invention, a storage prediction model for MPPDB data is obtained through machine learning training. Data changes in the MPPDB are predicted based on the storage prediction model, thereby optimizing and adjusting storage partitions in advance. This improves the query speed of the MPPDB, better meets user query and analysis needs, and reduces MPPDB storage resource investment.

Description

Technical Field

[0001] The present invention relates to the technical field of database management, and in particular to a data management method, a data management device and a storage medium. Background Art

[0002] With the widespread commercialization of 5G networks, the ubiquity of mobile internet, the sharp decline in data communication costs, and the emergence of various low-cost sensor technologies and smart devices, in addition to traditional mobile phones and computers collecting and generating real-time data, wristbands, shared bicycles, taxis, smart meters, elevators, and equipment on industrial production lines are all continuously generating real-time data. Massively Parallel Processing Databases (MPPDBs) can facilitate the management of this massive amount of data.

[0003] Under existing technology, database administrators need to regularly analyze the disk space status of MPPDB to maintain the normal operation of MPPDB under limited disk space conditions. However, database administrators have no way of knowing user query behavior, which data can be deleted due to expiration, which data cannot be deleted, and which data needs to be indexed to speed up user query efficiency. They can only manage according to human judgment. Therefore, the current MPPDB disk space management technology needs to be improved. Summary of the Invention

[0004] The following is a summary of the subject matter described in detail herein. This summary is not intended to limit the scope of the claims.

[0005] The embodiments of the present invention provide a data management method, a data management device, and a storage medium, which can optimize the occupancy of the disk space of an MPPDB.

[0006] In a first aspect, an embodiment of the present invention provides a data management method applied to a massively parallel processing database MPPDB, the data management method comprising:

[0007] Obtaining query behavior information of the user on the MPPDB based on the query request from the user;

[0008] Based on the query behavior information, a storage prediction model is generated through machine learning and a predicted storage state of the MPPDB is obtained according to the storage prediction model;

[0009] The storage partitions in the MPPDB are adjusted based on the predicted storage status.

[0010] In second aspect, an embodiment of the present invention provides a data management device, comprising at least one processor and a memory for communicating with the at least one processor; the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the aforementioned data management method.

[0011] In a third aspect, an embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute the data management method of the first aspect.

[0012] The data management method, data management device, and storage medium provided by the embodiments of the present invention obtain a storage prediction model for MPPDB data through machine learning training based on user query behavior on the MPPDB. Changes in data in the MPPDB are predicted based on the storage prediction model, thereby optimizing and adjusting storage partitions in advance, improving the query speed of the MPPDB, better meeting user query and analysis needs, and reducing MPPDB storage resource investment.

[0013] Other features and advantages of the present invention will be described in the following description, and in part will become apparent from the description, or will be understood by practicing the present invention. The purposes and other advantages of the present invention can be realized and obtained by the structures particularly pointed out in the description, claims and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The accompanying drawings are used to provide a further understanding of the technical solution of the present invention and constitute a part of the specification. Together with the examples of the present invention, they are used to explain the technical solution of the present invention and do not constitute a limitation to the technical solution of the present invention.

[0015] Figure 1 is a flow chart of a data management method provided by an embodiment of the first aspect of the present invention;

[0016] Figure 2 is a flow chart of a data management method provided by another embodiment of the first aspect of the present invention;

[0017] Figure 3 is a flow chart of a data management method provided by another embodiment of the first aspect of the present invention;

[0018] Figure 4 is a flow chart of a data management method provided by another embodiment of the first aspect of the present invention;

[0019] Figure 5 is a flow chart of a data management method provided by another embodiment of the first aspect of the present invention;

[0020] Figure 6 is a flow chart of a data management method provided by another embodiment of the first aspect of the present invention;

[0021] Figure 7 is a flow chart of a data management method provided by another embodiment of the first aspect of the present invention;

[0022] Figure 8 is a flow chart of a data management method provided by another embodiment of the first aspect of the present invention;

[0023] Figure 9 This is an example diagram of storage partitioning of a date in a practical example provided by the present invention;

[0024] Figure 10 This is an example diagram of storage partitions for another date of a practical example provided by the present invention;

[0025] Figure 11 It is a schematic diagram of the structure of a data management device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0026] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0027] An embodiment of the present invention provides a data management method, a data management device, and a storage medium, which utilize machine learning to perform training based on user query behavior information to obtain a storage prediction model. The storage prediction model is used to predict changes in storage partitions of a large-scale parallel processing database based on the user's query habits, thereby adjusting the storage partitions of the large-scale parallel processing database in advance, which can speed up the user's query speed in the large-scale parallel processing database.

[0028] The embodiments of the present invention are further described below with reference to the accompanying drawings.

[0029] The first aspect of the embodiment of the present invention provides a data management method, which is applied to a large-scale parallel processing database MPPDB, referring to Figure 1 The data management method of the embodiment of the present invention includes but is not limited to the following steps S100, S200 and S300.

[0030] Step S100 : obtaining the query behavior information of the user querying the MPPDB based on the query request from the user.

[0031] MPPDB is used to store structured data. The so-called structured data refers to data that is logically expressed and implemented through a two-dimensional table structure, including data generated by sensors, terminal devices, transaction systems, etc. These data usually have, but are not limited to, the following characteristics: (1) The data is time-series and must have a timestamp; (2) The data has a retention period; (3) Most of the data contains geographic location information; (4) The data is rarely updated or automatically deleted; (5) The data volume is huge, and the amount of data collected and reported in a day may exceed 10 million; (6) The query and analysis of the data is based on time period and geographic location information. Therefore, in MPPDB, these data are stored in the form of two-dimensional tables. In order to speed up the query, MPPDB divides the two-dimensional table into multiple storage partitions. Each storage partition can be distinguished based on time period or geographic location information. When the database receives a query initiated by a user, MPPDB initiates a query on multiple storage partitions in parallel based on its own query method. Finally, it summarizes the query sub-results of each storage partition and returns the result of this query to the user. In order to improve the accuracy of queries in the database, users' query requests usually include one or more information such as the table involved in the query, partition information, index status, data volume, time span, and geographic location. Based on this, step S100 of the embodiment of the present invention obtains various information included in the user query request. By parsing the various information in the user query request, the user's query behavior information on the MPPDB can be obtained, thereby providing feature information for model training for the machine learning embodiment of the present invention. It can be understood that the above-mentioned user query can be one or more query requests initiated by one or more users, and these query requests can be initiated by one or more client devices to the MPPDB via local or network means, and the client can be any device requesting to execute a query, such as a computer terminal, a smart phone, etc.

[0032] Step S200 : Based on the query behavior information, a storage prediction model is generated through machine learning and a predicted storage state of the MPPDB is obtained according to the storage prediction model.

[0033] From the several characteristics of the data described above, it can be seen that as time goes by, the data in the database will become more and more, while the disk space of the MPPDB is limited. The database administrator needs to regularly analyze the disk space status of the data. If the disk space usage is found to be abnormal, the data in the database needs to be manually adjusted. However, the database administrator cannot accurately understand the user's query behavior and cannot accurately manage the data in the database. Based on this, the embodiment of the present invention uses machine learning technology to train the user's query behavior information obtained by parsing in step S100, establishes the correlation between the query behavior information and the storage method of the data in the MPPDB, and thus obtains a storage prediction model. Through this storage prediction model, the storage status of the data in the disk can be predicted, such as the amount of data written, the number of times the data is queried, the valid time of the data, and the location information classification of the data. The MPPDB automatically manages the data (or storage partition) in the database based on the predicted storage status obtained by the storage prediction model, thereby optimizing the storage method of the data in the MPPDB. It can be understood that the implementation of the prediction training model through machine learning technology in the embodiment of the present invention actually includes the process of characterizing the query behavior information, that is, extracting feature information suitable for the machine learning training model based on the user's query behavior, and the above-mentioned storage prediction model can be obtained based on the feature information and the training model. It is understandable that the implementation of machine learning technology is based on one or more existing algorithms, such as decision tree algorithm, artificial neural network algorithm, deep learning, etc. These algorithms have been widely used in big data processing. The specific algorithm selection is not the focus of this invention. In order to save space, the implementation method of the training model through the algorithm will not be expanded here.

[0034] Step S300: adjusting the storage partitions in the MPPDB based on the predicted storage status.

[0035] It is understandable that the data in the database can be automatically managed according to the predicted storage status. The optimization strategy pre-stored in MPPDB can be called to adjust the data (or storage partition), or the data (or storage partition) can be adjusted according to the actual predicted storage status in real time. The adjustment methods involved include but are not limited to: creating / deleting the index of the storage partition, reducing the level of the index, creating / deleting the storage partition, deleting expired data, etc. These specific adjustment methods will be further explained below. The above adjustment process does not require the participation of the database administrator. The above adjustment process can be triggered according to the passage of time, or through the query initiated by the user, and ultimately the data storage method in MPPDB is optimized, thereby reducing the frequency of data loading and improving the query speed. Through automatic adjustment, the disk occupancy rate of MPPDB can also be maintained within a reasonable range, reducing disk overhead.

[0036] In one embodiment, referring to Figure 2 , step S300 includes the following step S310:

[0037] Step S310: Create a first storage partition in the MPPDB according to the predicted storage status.

[0038] As can be seen from the above description, the storage prediction model is used to predict the changes in data in MPPDB. Then, when the system time enters the next time point (for example, the system time comes to a new day) or receives a user's query request, the storage prediction model predicts the disk space of MPPDB based on the time point or the user's query; for example, when the system time comes to a new day, based on the previous data write volume prediction, it is necessary to create a new storage partition in MPPDB to save the data to be written, and according to the size of the data write volume, set a time range for the newly created storage partition (data with a timestamp within the set time range are all written to this storage partition). By creating the first storage partition in advance, delays caused by creating partitions when new data is written can be prevented. It is worth noting that the time range of the first storage partition created in advance can be beyond the current time, that is, a storage partition for a future date is created under the current date, and as the date continues to advance, new storage partitions are continuously created, thereby achieving rapid storage of reported data.

[0039] In one embodiment, referring to Figure 3 The query behavior information in step S200 includes the user's query granularity, and the predicted storage status includes the predicted data write amount of the first storage partition. Based on the above step S310, step S300 further includes the following step S320:

[0040] Step S320: Determine the time range of the first storage partition based on the predicted data write amount and the query granularity.

[0041] There are two main factors that determine the time range of the newly created first storage partition. One is the query granularity of the user's data query. That is, data within a certain time period is frequently queried by users. To improve query speed, the first storage partition is created based on this time period (i.e., time range). In this way, based on the user's query habits, MPPDB can quickly locate the first storage partition, thereby speeding up the query results. The other is the predicted data write volume. As mentioned above, the amount of data reported in MPPDB is large and shows obvious temporal patterns. That is, the amount of data reported in some time periods is greater than that in other time periods. For example, between dusk and late at night, the amount of data generated by smartphone terminals is very large, even exceeding the amount of data reported in other time periods of the day. The data in this time period is often more valuable for enterprises to analyze. Therefore, to facilitate enterprises / users to search and analyze data in this time period, the time range of the first storage partition is set according to this time period. Of course, in this case, since the amount of data reported in other time periods is not large, the time range of the first storage partition can also be directly set according to one or more consecutive natural days, for example, setting the time range of the first storage partition to April 1, 2020.

[0042] In one embodiment, referring to Figure 4 The query behavior information in step S200 includes the query frequency of the user querying different storage partitions. Based on the first storage partition created in step S310, the data management method of the embodiment of the present invention further includes steps S400 and S410:

[0043] Step S400: Based on the query frequency, obtain the predicted query frequency of the user for the first storage partition according to the storage prediction model;

[0044] Step S410: When the predicted query frequency is greater than the query frequency threshold, an index is created for the first storage partition.

[0045] This embodiment analyzes the user's query frequency for recent data. If it is higher than the query frequency threshold, it is considered that there is a high possibility of frequent queries on the first storage partition. Therefore, in order to speed up the query speed of the first storage partition, an index is created for the first storage partition. The index in MPPDB is a structure that sorts the values ​​of one or more columns in the database table. The use of the index can quickly access specific information in the database table, that is, compared with searching all rows in the table, the index helps to obtain information faster. It is worth noting that the query frequency threshold mentioned in this embodiment is used to distinguish whether the user query has reached a frequent level. This threshold can be a frequency threshold set manually based on experience, or it can be a frequency threshold derived by the storage prediction model based on the user's query behavior information.

[0046] In one embodiment, referring to Figure 5 If the query behavior information in step S200 includes the query time range of the user's query on the data in the MPPDB, then step S300 further includes the following step S330:

[0047] Step S330: Delete the data in the storage partition that is out of the query time range.

[0048] This embodiment takes into account the processing of expired data in MPPDB. Expired data generally refers to discarded data that has no query value, such as the operating parameters of some old equipment after equipment upgrades, call records with a certain age, etc. Enterprises / users no longer query these data, so database administrators are often required to manually find and delete these data. For example, some data with a high update frequency has no reference value when it exceeds 30 days, and is beyond the time range for enterprises / users to query it. Therefore, the storage prediction model deletes the data in the storage partition based on the query time range in the query behavior information, which can reduce the storage pressure of MPPDB and avoid the lag of manual maintenance.

[0049] In one embodiment, referring to Figure 6 and Figure 7 , the above step S330 also includes step S331 or step S332:

[0050] Step S331: When the timestamps of the data in the storage partition are all out of the query time range, the storage partition is deleted.

[0051] Step S332: When the timestamp of a part of the data in the storage partition exceeds the query time range, the storage partition is split into an overdue partition and a non-overdue partition, and the overdue partition is deleted, wherein the timestamps of the data in the overdue partition are all beyond the partition of the query time range, and the timestamps of the data in the non-overdue partition are all within the partition of the query time range.

[0052] Step S331 and step S332 are two different strategies for deleting data in storage partitions. Since storage partitions can be set with different time ranges, in the process of deleting data by time, it is inevitable that part of the data in a storage partition is overdue data, while the remaining data is non-overdue data. Under the strategy of step S331, waiting for all data in the storage partition to expire before deleting the storage partition can reduce the operations on the storage partition to a certain extent. Under the strategy of step S332, overdue data and non-overdue data are split into two storage partitions, and only the storage partition with overdue data is deleted. In this way, the time range of the original storage partition can be ignored, and data can be deleted in real time to reduce storage pressure, but the operations on the storage partition are more than those of the strategy of step S331.

[0053] In one embodiment, referring to Figure 8 The query behavior information in step S200 includes the query frequency of the user querying different storage partitions and the query time range of the user querying the data in the MPPDB; then, the data management method of the embodiment of the present invention further includes but is not limited to the following steps S510, S520, S530 and S540:

[0054] Step S510: when the occupancy rate of the MPPDB storage space is lower than a first occupancy rate threshold, deleting data in the storage partition that exceeds the query time range;

[0055] Step S520: When the occupancy rate of the MPPDB storage space is higher than the first occupancy rate threshold and lower than the second occupancy rate threshold, delete the index of the storage partition whose predicted query frequency is lower than the query frequency threshold;

[0056] Step S530: When the occupancy rate of the MPPDB storage space is higher than the second occupancy rate threshold and lower than the third occupancy rate threshold, performing a partition exchange operation on the storage partition to compress the data of the storage partition;

[0057] Step S540 : When the occupancy rate of the MPPDB storage space is higher than the third occupancy rate threshold, data in the storage partition whose predicted query frequency is lower than the query frequency threshold is deleted.

[0058] This embodiment proposes an optimization strategy based on an autonomous multi-level feedback mechanism. Through steps S510 to S540, the disk space of the MPPDB can be autonomously maintained. For example, a first occupancy threshold is set to 25%, a second occupancy threshold is set to 50%, and a third occupancy threshold is set to 80%. When the disk occupancy of the MPPDB is lower than 25%, even if the user does not perform any operations on the MPPDB, the optimization strategy based on the autonomous multi-level feedback mechanism deletes data in the storage partitions that exceed the query time range based on the query time range. For example, by scanning the time of each storage partition on the disk, it is determined whether the query time range is exceeded. If any data in the storage partition exceeds the query time range, the expired data is deleted or the storage partition is even deleted. It is understandable that the disk scanning time can be set as needed. If the data is updated quickly and the query time range is small in granularity, a shorter scanning time can be set. If the data is updated slowly and the query time range is large in granularity, such as 60 days, a longer scanning time can be set, such as once a day. When the disk occupancy rate of MPPDB is higher than 25% and lower than 50%, the indexes of some storage partitions are deleted. These storage partitions are partitions that the storage prediction model infers that users will not frequently query (i.e., partitions below the query frequency threshold). Since indexes also occupy a certain amount of storage space, the indexes of storage partitions below the query frequency threshold can be deleted to reduce the disk occupancy rate. When the disk occupancy rate of MPPDB is higher than 50% and lower than 80%, the disk occupancy rate is reduced by swapping partitions. During the query process, the storage partitions may expand due to different degrees of data loading. At this time, the partition swap operation can compress the data in the expanded partitions, thereby reducing the disk occupancy rate. When the disk occupancy rate of MPPDB is higher than 80%, it is considered that the disk occupancy is quite high and more radical means are needed to process the data. In this embodiment, all data that users do not frequently query is deleted and only the data that users frequently query is retained, thereby significantly reducing the data on the disk. It is understandable that the above-mentioned occupancy thresholds can be set according to actual needs, and the optimization strategy corresponding to the interval formed by each occupancy threshold can also be set as needed. This embodiment is only explained in an illustrative manner, and the purpose is to illustrate that the use of the above-mentioned optimization strategy can reduce disk occupancy, ensure that the MPPDB system is in an available state, and provide users with better query services.

[0059] In one embodiment, the storage prediction model is generated by machine learning in step S200, including: using the query behavior information as a training set, training the query behavior information by machine learning based on the stacking method, and obtaining the storage prediction model.

[0060] To ensure the accuracy and robustness of the stored prediction model, the training model is trained using a stacking training method. Existing methods are prone to inconsistencies between validation and test errors in practical applications. This inconsistency is caused by overfitting and concept drift, as well as the strong correlation of the base classifier model. For example, in user query behavior information, query frequency is often related to the attributes of the query object, meaning that users tend to frequently query attributes that they are most interested in. To address this issue, a multi-level stacking technique is used to train a robust system with consistent validation and test performance. An embodiment of the present invention provides a schematic diagram of a two-level stacking technique. The training data is divided into five layers, and five different models are trained using a leave-one-out strategy. This strategy uses one of the five layers of training data for prediction, while the remaining four layers are used for training. The trained models are then used for testing, generating a prediction for each model. To predict the test data, the average of each model's prediction scores is determined, and the average of the five prediction scores forms a sixth model. The five prediction scores are then treated as new training data on the test data. The new training and test datasets of the prediction model are used to generate the final result, i.e., the stored prediction model. The advantage of the stacking method is that it avoids overfitting caused by cross-validation in traditional training methods and explains the nonlinearity between features caused by taking the output as a feature.

[0061] The above embodiment establishes a storage prediction model through machine learning based on the user's behavioral habits of querying data in the MPPDB, as well as the data circulation and storage status of the current database. The MPPDB's disk space is automatically managed according to the storage prediction model, thereby optimizing the data storage method. Compared with manual analysis and management by database administrators, the embodiment of the present invention can obtain analysis results more accurately, thereby reducing disk overhead.

[0062] The embodiments of the present invention are described below with practical examples:

[0063] Reference Figure 9In a target table of a specific measurement object, there are seven storage partitions, namely 20200501_7, 20200508_7, 20200515_3, 20200518_3, 20200521_3, 20200524_3, and 20200527_3 (in this example, the storage partitions are named by date_day). It is also set that the user frequently queries the data of the latest week in MPPDB (that is, the data greater than the query frequency threshold is the data of the latest week), and the user is set to query only the data of the latest three weeks (that is, The query time range is three weeks); assuming the current time is May 27, 2020, the storage prediction model learns from the user's query behavior information that the user will frequently query the data of the past week. At the same time, it is also known that the data reporting volume of MPPDB in the next few days is medium. Therefore, a new partition 20200530_3 is automatically created, and an index is created for 20200530_3. On the other hand, since 20200501_7 has exceeded the query time range of three weeks, the storage partition 20200501_7 is deleted; refer to Figure 10 As time passes, the date reaches May 30, 2020. Based on previous user query habits, users will still frequently query data from the past week. However, the amount of data reported on the last day of the month is very large. Therefore, a new partition 20200531_1 is created and an index is created for it. At the same time, because the storage partition 20200521_3 exceeds the time range of a week, the index of 20200521_3 is deleted. By creating a storage partition with a time range of one day, only the data for May 31st is stored, meeting the user's fine-grained query needs to be faster and more reliable. In actual scenarios, the time range of creating a new partition needs to refer to multiple dimensions. For example, data from different areas has different data attributes. In an area divided by base station coverage, data for the current area is reported at a granularity of 15 minutes and retained for 30 days. However, in an area covered by another base station adjacent to the current area, due to the overlapping coverage of the two base stations, data from the adjacent area may enter the current area. Although the data of the neighboring area is also reported at a granularity of 15 minutes, it is only retained for 7 days.

[0064] A second aspect of an embodiment of the present invention provides a data management device, comprising at least one processor and a memory for communicating with the at least one processor; the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the data management method of the first aspect mentioned above.

[0065] Reference Figure 11, taking the example that the control processor 1001 and the memory 1002 in the data management device 1000 can be connected via a bus. The memory 1002, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer executable programs. In addition, the memory 1002 may include a high-speed random access memory, and may also include a non-transitory memory, such as at least one disk memory, a flash memory device, or other non-transitory solid-state memory device. In some embodiments, the memory 1002 may optionally include a memory remotely arranged relative to the control processor 1001, and these remote memories may be connected to the data management device 1000 via a network. Examples of the above-mentioned networks include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0066] Those skilled in the art will understand that Figure 11 The device structure shown in the figure does not constitute a limitation to the data management device 1000, and the data management device 1000 may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0067] The data management device of the embodiment of the present invention executes the data management method of the first aspect, establishes a storage prediction model through machine learning based on the user's behavioral habits of querying data in the MPPDB, as well as the data circulation and storage status of the current database, and automatically manages the disk space of the MPPDB according to the storage prediction model, thereby optimizing the data storage method. Compared with the manual analysis and management of the database administrator, the embodiment of the present invention can obtain analysis results more accurately, thereby reducing disk overhead.

[0068] A third aspect of an embodiment of the present invention provides a computer-readable storage medium storing computer-executable instructions, which are executed by one or more control processors, for example, Figure 11 The execution of one of the control processors 1001 in the embodiment of the present invention may cause the one or more control processors to execute the data management method in the embodiment of the present invention, for example, executing the data management method described above. Figure 1 Steps S100 to S300 of the method, Figure 2 Method step S310, Figure 3 Method step S320, Figure 4 Steps S410 to S420 of the method, Figure 5 Method step S330, Figure 6 Step S331 of the method, Figure 7 Method step S332 and Figure 8 Method steps S510 to S540.

[0069] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, i.e., they may be located in one place or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of this embodiment.

[0070] Those skilled in the art will appreciate that all or some of the steps and systems in the method disclosed above can be implemented as software, firmware, hardware, and appropriate combinations thereof. Some physical components or all physical components can be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or implemented as hardware, or implemented as an integrated circuit, such as an application-specific integrated circuit. Such software can be distributed on a computer-readable medium, and the computer-readable medium can include computer storage media (or non-transitory media) and communication media (or temporary media). As known to those skilled in the art, the term computer storage media is included in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data) and is volatile and non-volatile, removable, and non-removable. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory, or other memory technology, CD-ROM, digital versatile disks (DVD), or other optical disk storage, magnetic cassettes, magnetic tapes, disk storage, or other magnetic storage devices, or any other medium that can be used to store desired information and can be accessed by a computer. Furthermore, as is well known to those skilled in the art, communication media typically embodies computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

[0071] The above is a specific description of the preferred implementation of the present application, but the present application is not limited to the above implementation mode. Technical personnel familiar with the field can also make various equivalent modifications or substitutions without violating the spirit of the present application. These equivalent modifications or substitutions are all included in the scope defined by the claims of the present application.

Claims

1. A data management method, applied to a massively parallel processing database (MPPDB), comprising: Obtaining query behavior information of the user querying the MPPDB based on a query request from the user, wherein the query behavior information includes the query granularity of the user; Based on the query behavior information, a storage prediction model is generated through machine learning and a predicted storage state of the MPPDB is obtained according to the storage prediction model; Creating a first storage partition in the MPPDB according to the predicted storage state, wherein the predicted storage state includes a predicted data write amount of the first storage partition; Determining a time range of the first storage partition according to the predicted data write amount and the query granularity; The query granularity reflects whether the data within a certain time period is frequently queried by users; data with timestamps within the time range are all written to the first storage partition, and the time range includes time exceeding the current time.

2. The data management method according to claim 1, wherein: The query behavior information also includes the query frequency of users querying different storage partitions; The data management method further includes: Based on the query frequency, obtaining a predicted query frequency of the user for the first storage partition according to the storage prediction model; When the predicted query frequency is greater than a query frequency threshold, an index is created for the first storage partition.

3. The data management method according to claim 1, wherein: The query behavior information also includes the query time range of the user's query on the data in the MPPDB; The data management method further includes: Delete data in the storage bucket that is outside the query time range.

4. The data management method according to claim 3, wherein: Deleting data in the storage partition that is beyond the query time range includes one of the following: When the timestamps of the data in the storage partition are all out of the query time range, the storage partition is deleted; When the timestamp of a portion of data in a storage partition exceeds the query time range, the storage partition is split into an overdue partition and a non-overdue partition, and the overdue partition is deleted, wherein the timestamps of the data in the overdue partition all exceed the partition of the query time range, and the timestamps of the data in the non-overdue partition all do not exceed the partition of the query time range.

5. The data management method according to claim 1, wherein: The query behavior information also includes the query frequency of users querying different storage partitions and the query time range of users querying data in MPPDB; The data management method further includes: When the occupancy rate of the MPPDB storage space is lower than a first occupancy rate threshold, deleting data in the storage partition that exceeds the query time range; When the occupancy rate of the MPPDB storage space is higher than the first occupancy rate threshold and lower than the second occupancy rate threshold, deleting the index of the storage partition whose predicted query frequency is lower than the query frequency threshold; When the occupancy rate of the MPPDB storage space is higher than the second occupancy rate threshold and lower than the third occupancy rate threshold, performing a partition swap operation on the storage partition to compress data in the storage partition; When the occupancy rate of the MPPDB storage space is higher than the third occupancy rate threshold, the data in the storage partition whose predicted query frequency is lower than the query frequency threshold is deleted.

6. The data management method according to claim 5, characterized in that: The partition exchange operation on the storage partition includes: Identify the storage partitions where data expansion occurs; Create a new second storage partition, copy the data of the storage partition with data expansion to the second storage partition and compress it; Update the partition association information of the MPPDB and delete the storage partition where data expansion occurs.

7. The data management method according to claim 1, wherein: Generating a storage prediction model through machine learning includes: The query behavior information is used as a training set, and based on the stacking method, the query behavior information is trained through machine learning to obtain a storage prediction model.

8. A data management device, characterized in that: It includes at least one processor and a memory for communicating with the at least one processor; the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the data management method as described in any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute the data management method according to any one of claims 1 to 7.

Images