Data storage method, electronic device, and computer-readable storage medium

By dynamically adjusting data table partitions, the problem of unpredictable data table partition sizes in SaaS services is solved, achieving reasonable partitioned storage, reducing I/O waste, and improving the efficiency of analysis services.

CN112328592BActive Publication Date: 2024-11-29HUAWEI TECH CO LTD
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202011104762.8
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2020-10-15
Publication Date
2024-11-29
Estimated Expiration
2040-10-15

AI Technical Summary

Technical Problem

In existing SaaS services, the size of data table partitions cannot be estimated reasonably, resulting in wasted I/O resources and reduced response speed of analysis services.

Method used

By determining the time information of the data to be stored and its matching with the nearest table partition and storage space, new table partitions are dynamically created or merged, ensuring that the data table partitions are stored at a reasonable size and reducing system I/O waste.

Benefits of technology

This system enables the matching of table partition storage data time information with its own time information, reducing system I/O waste and improving the response speed and stability of analysis services.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN112328592B_ABST
    Figure CN112328592B_ABST
Patent Text Reader

Abstract

The embodiments of the present application provide a data storage method, an electronic device, and a computer-readable storage medium, and relate to the field of computer technology. When there is a storage demand for transferred data, by obtaining the time information of the most recent table partition of the data table in the database and comparing it with the time information of the data to be transferred, and comparing the remaining storage space of the most recent table partition with the data amount of the data to be transferred, it is determined whether the data to be transferred meets the storage condition of being stored in the most recent table partition. When the storage condition is met, the data to be transferred is stored in the most recent table partition. When the storage condition is not met, a new table partition is created in the data table, and the data to be transferred is stored in the newly created table partition. The present application determines whether to place the newly-incoming data in the data table in a newly-created table partition or an existing table partition, and makes the data stored in each table partition in the data table as close to a reasonable partition size as possible, which can minimize the waste of system I / O.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a data storage method, an electronic device, and a computer-readable storage medium. Background Art

[0002] With the development of Internet technology and the maturity of application software, an innovative software application model, Software-as-a-Service (SaaS), has begun to emerge. Third-party vendors can obtain corresponding software services by purchasing SaaS services. In existing SaaS services, the data analysis model provided by the SaaS service provider is preset, and the algorithm of the analysis indicator is also preset. Multiple third-party vendors share the same set of data resources for data analysis. This set of data resources collects data to be analyzed from multiple third-party vendors.

[0003] like Figure 1 As shown, multiple third-party manufacturers operate multiple applications (Application, APP) APP1~APP i , the data of multiple APPs are stored in multiple databases Db1~Db j . There are generally two ways to store the data of multiple apps in data tables of multiple databases: the first is to perform a hash operation based on the APP identification information APP_ID and perform a modulus operation on the hash operation result to find the corresponding database, and then store the APP data in the data table of the database; the second is to pre-establish a mapping relationship between the APP identification information APP_ID and the database identification information Database_ID, and then store the APP data in the data table of the corresponding database according to the mapping relationship. Since the analysis services provided by SaaS generally need to be analyzed based on the timeline, the data table often needs to be partitioned according to time. Since the data of each APP cannot be estimated, there is a problem of being unable to estimate the partition size when partitioning the data table. If there are too many table partitions with small partitions in the data table, a large amount of resources will be consumed in the establishment of input / output (I / O) when reading the table partition data later. Existing analysis services generally respond in seconds. Simply partitioning the data table by time cannot make the data stored in the table partition close to a reasonable size, which reduces the response speed of the analysis service. Summary of the invention

[0004] In view of this, it is necessary to provide a data storage method, an electronic device and a computer-readable storage medium, so that the data stored in the table partition is close to a reasonable size and the system I / O waste is minimized.

[0005] A first aspect of an embodiment of the present application discloses a data storage method, which is applied to a database, wherein the database includes at least one data table, and the data storage method includes: in response to the storage demand of the data to be transferred, determining the data table in the database corresponding to the data to be transferred; obtaining the time information and remaining storage space of the most recent table partition of the data table, and the time information and data volume of the data to be transferred; and if the time information of the most recent table partition does not match the time information of the data to be transferred and / or the remaining storage space of the most recent table partition is less than the data volume of the data to be transferred, creating a new table partition in the data table, and storing the data to be transferred in the newly created table partition.

[0006] By adopting this technical solution, for each extracted data to be transferred, when the table partition time information does not match the time information of the data to be transferred and / or the remaining storage space of the table partition is less than the data amount of the data to be transferred, a new table partition is created for storing the data to be transferred, so that the data stored in the table partition is as close to the reasonable partition size as possible, and the time information of the data stored in the table partition matches its own time information, which can minimize the system I / O waste and facilitate subsequent data analysis.

[0007] In one possible implementation, the data to be transferred includes data attributes, each data table in the database includes table attributes, and the table attributes are used to indicate data attributes of data that can be stored in the data table. Determining the data table in the database corresponding to the data to be transferred includes: determining the data table corresponding to the data to be transferred based on the data attributes of the data to be transferred and the table attributes of each of the data tables in the database.

[0008] By adopting this technical solution, it is possible to determine the data table corresponding to the data to be transferred each time, so as to facilitate the subsequent judgment of whether the data to be transferred is placed in a newly created table partition or an existing table partition in the corresponding data table.

[0009] In a possible implementation, the data to be transferred is stored in a first server, the database is deployed in a second server, and the method further includes: extracting the data to be transferred from the first server using a preset data extraction tool, and sending data attributes of the data to be transferred to the second server.

[0010] By adopting this technical solution, the data stored in the first server can be transferred to the second server, so that the second server can subsequently provide designated service functions.

[0011] In one possible implementation, the method further includes: setting the time granularity of the table partition of the data table according to the service level agreement (SLA) of the second server; setting the partition size of the table partition of the data table according to the hardware configuration information of the second server, wherein the hardware configuration information includes at least disk read / write (I / O) performance.

[0012] By adopting this technical solution, it is possible to set the time granularity of the table partition based on the SLA of the analysis service provided by the second server, and to set the table partition to a reasonable partition size according to the hardware resources of the second server itself.

[0013] In a possible implementation, the method further includes: if the time information of the most recent table partition and the time information of the data to be transferred do not belong to the same time granularity, determining that the time information of the most recent table partition does not match the time information of the data to be transferred.

[0014] By adopting this technical solution, it is possible to determine whether the time information of the most recent table partition matches the time information of the data to be transferred based on the time granularity of the table partition.

[0015] In a possible implementation, the time information of the data to be transferred includes a start time and an end time, the amount of data to be transferred extracted by the preset data extraction tool from the first server each time is smaller than the partition size of the table partition, and the start time and end time of the data to be transferred belong to the same time granularity.

[0016] By adopting this technical solution, it is possible to avoid the situation where the data to be transferred cannot be stored in a newly created table partition or an existing table partition in the data.

[0017] In a possible implementation, the method further includes: if the time information of the most recent table partition matches the time information of the data to be transferred and the remaining storage space of the most recent table partition is greater than or equal to the data volume of the data to be transferred, storing the data to be transferred in the most recent table partition.

[0018] By adopting this technical solution, it is possible to determine whether to place the data to be transferred in a newly created table partition or an existing table partition for each extraction, so that the data stored in the table partition is as close as possible to the reasonable partition size, which can minimize the system I / O waste.

[0019] In a possible implementation, the time information of the data to be transferred and the most recent table partition both include a start time and an end time. After storing the data to be transferred in the most recent table partition, it also includes: if the end time of the data to be transferred is greater than the end time of the most recent table partition, updating the end time of the most recent table partition to the end time of the data to be transferred.

[0020] By adopting this technical solution, it is possible to update the end time of the most recent table partition according to the end time of the data to be transferred.

[0021] In a possible implementation, the acquiring time information of the most recent table partition of the data table includes: acquiring the start time and the end time of the most recent table partition of the data table by using a preset database query statement.

[0022] By adopting this technical solution, it is possible to obtain the time information of the most recent table partition of a data table by using a database query statement.

[0023] In a possible implementation manner, creating a new table partition in the data table includes: creating the new table partition in the data table based on time information of the data to be transferred.

[0024] By adopting this technical solution, it is possible to create a new table partition in the data table based on the time information of the data to be transferred, so that the time information of the table partition corresponds to the time information of the data to be transferred.

[0025] In a possible implementation, the data storage method further includes: determining n table partitions in the data table that can be partitioned and merged according to a preset partition merging rule, wherein the n table partitions are arranged in sequence in chronological order, and the preset partition merging rule defines a partition threshold for merging; sequentially attempting to merge m adjacent table partitions starting with each of the table partitions, wherein m and n are both positive integers, and m is less than n; if the sum of the partition sizes of the m adjacent table partitions is less than or equal to the partition threshold, merging the m adjacent table partitions into a new table partition, and continuing to attempt to merge the m adjacent table partitions starting with the new table partition; if the sum of the partition sizes of the m adjacent table partitions is greater than the partition threshold, abandoning the partition merging of the m adjacent table partitions.

[0026] By adopting this technical solution, table partitions can be merged into table partitions that are more suitable for analysis or query services, which can reduce the hardware resource occupancy when responding to analysis or query services and shorten the response time of analysis or query services.

[0027] In a possible implementation, the preset partition merging rule also defines merging time information, and determining the n table partitions in the data table that can be partitioned and merged according to the preset partition merging rule includes: filtering out table partitions from the data table whose time information falls within the merging time information.

[0028] By adopting this technical solution, it is possible to filter out table partitions that can be merged from a specified data table.

[0029] In a possible implementation manner, the database is deployed on a second server, and the method further includes: setting the merging time information according to an SLA of the second server.

[0030] By adopting this technical solution, it is possible to dynamically adjust the table partition size based on the time granularity of the service provided by the server.

[0031] A second aspect of an embodiment of the present application discloses a data storage method, which is applied to a database, wherein the database includes at least one data table, and the data storage method includes: determining n table partitions in the data table that can be partitioned and merged according to a preset partition merging rule, wherein the n table partitions are arranged in sequence in chronological order, and the preset partition merging rule defines a partition threshold for merging; sequentially attempting to merge m adjacent table partitions starting with each of the table partitions, wherein m and n are both positive integers, and m is less than n; if the sum of the partition sizes of the m adjacent table partitions is less than or equal to the partition threshold, merging the m adjacent table partitions into a new table partition, and continuing to attempt to merge the partitions of the m adjacent table partitions starting with the new table partition; if the sum of the partition sizes of the m adjacent table partitions is greater than the partition threshold, abandoning the partition merging of the m adjacent table partitions.

[0032] By adopting this technical solution, table partitions can be merged into table partitions that are more suitable for analysis or query services, which can reduce the hardware resource occupancy when responding to analysis or query services and shorten the response time of analysis or query services.

[0033] In a third aspect, an embodiment of the present application provides a computer-readable storage medium, including computer instructions. When the computer instructions are executed on an electronic device, the electronic device executes the data storage method described in the first aspect or the second aspect.

[0034] In a fourth aspect, an embodiment of the present application provides an electronic device, comprising a processor and a memory, wherein the memory is used to store instructions, and the processor is used to call the instructions in the memory so that the electronic device executes the data storage method described in the first aspect or the second aspect.

[0035] In a fifth aspect, an embodiment of the present application provides a computer program product, which, when executed on a computer, enables the computer to execute the data storage method as described in the first aspect or the second aspect.

[0036] In a sixth aspect, a device is provided, which has the function of implementing the behavior of the first electronic device in the method provided in the first aspect or the second aspect. The function can be implemented by hardware, or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions.

[0037] It can be understood that the computer-readable storage medium described in the third aspect, the electronic device described in the fourth aspect, the computer program product described in the fifth aspect, and the device described in the sixth aspect all correspond to the methods of the first aspect or the second aspect above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding methods provided above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 A schematic diagram of a scenario for storing data of multiple APPs in the prior art.

[0039] Figure 2 This is an application scenario diagram of the data storage method provided in one embodiment of the present application.

[0040] Figure 3 A schematic diagram of interaction for data exchange between a first server and a second server provided in an embodiment of the present application.

[0041] Figure 4 A schematic diagram of a scenario for creating table partitions in a data table provided in an embodiment of the present application.

[0042] Figure 5 A flowchart of a data storage method provided in one embodiment of the present application.

[0043] Figure 6 A schematic diagram of interaction for data exchange between a data table management device and an analysis database provided in one embodiment of the present application.

[0044] Figure 7 A schematic diagram of a scenario for merging table partitions in a data table provided in an embodiment of the present application.

[0045] Figure 8 A flowchart of a data storage method provided in Embodiment 01 of the present application.

[0046] Fig. 9 A schematic diagram of the structure of a possible second server provided in an embodiment of the present application. DETAILED DESCRIPTION

[0047] It should be noted that in this application, "at least one" means one or more, and "more than one" means two or more than two. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The terms "first", "second", "third", "fourth", etc. (if any) in the specification, claims and drawings of this application are used to distinguish similar objects, rather than to describe a specific order or sequence.

[0048] Combine the following Figure 2 An application scenario diagram of the data storage method provided in an embodiment of the present application is exemplified.

[0049] The first storage area 100 is a storage area of ​​the first server 200 , and the second storage area 110 is a storage area of ​​the second server 210 . The first server 200 and the second server 210 can communicate with each other through a network.

[0050] The first storage area 100 is used to store source data, which may refer to data to be analyzed, such as user data and login data of applications (Application, APP) developed by multiple third-party manufacturers, user data and browsing data of websites developed by multiple third-party manufacturers, etc.

[0051] The first storage area 100 can be a storage area on a server owned by a third-party manufacturer, or a storage area on other servers that do not belong to the third-party manufacturer (intermediary server). The server of the third-party manufacturer can upload the recorded operation data to the intermediary server regularly or irregularly.

[0052] The source data stored in the first storage area 100 can be imported into the second storage area 110 through a data storage tool. The second storage area 110 is constructed with a database (a database is a collection of data tables, and the data tables are used to store data), such as the analysis database 12. The source data stored in the first storage area 100 can be stored in the analysis database 12. The data in the analysis database 12 can be used to provide data analysis services. For example, the second storage area 110 is the data storage area of ​​the analysis platform, and the APP data collected by the third-party manufacturer is imported into the analysis database of the analysis platform. The operating personnel of the third-party manufacturer can perform big data analysis on the APP data collected by the third-party manufacturer on the analysis platform to adjust the market strategy according to the analysis results, or optimize the APP to improve the user experience, etc. The analysis platform can be implemented based on an online analytical processing (ROLAP) engine based on a relational database, or based on other data analysis engines, which is not limited here.

[0053] Optionally, in some embodiments of the present application, the data storage tool may be an Extract Transform and Load (ETL) tool, or may be other storage tools / plug-ins that can transfer the source data stored in the first storage area 100 to the second storage area 110 .

[0054] Optionally, in some embodiments of the present application, both the first storage area 100 and the second storage area 110 may be a hard disk of a server. The database constructed on the second storage area 110 may not be an analysis database, but a database that meets other requirements (non-analysis services).

[0055] The second server 210 is also provided with a data table management device 11 and an ETL tool 13. The data table management device 11 can perform partition management on the data table in the analysis database 12, such as creating a new table partition in the data table, merging the original table partitions, etc. The ETL tool 13 can import the data stored in the first storage area 100 into the analysis database 12.

[0056] When the ETL tool 13 obtains the source data of the data table to be stored in the analysis database 12 from the first storage area 100, the ETL tool 13 will communicate with the data table management device 11 to obtain the judgment result of whether it is necessary to add a new table partition to the data table. If the data table management device 11 determines that a new table partition needs to be added to the data table, the ETL tool 13 will import the obtained source data into the newly added table partition in the data table. If the data table management device 11 determines that it is not necessary to add a new table partition to the data table, the ETL tool 13 will import the obtained source data into the existing table partition in the data table.

[0057] In some embodiments of the present application, the analysis database 12 may include multiple data tables, each of which may correspond to different business function data. For example, the analysis database 12 includes a login data table and a browse data table. The login data table is used to store login data, and the browse data table is used to store browse data. The data table management device 11 is used to partition the data table, and multiple table partitions can be added under the data table according to the specified time granularity. For example, table partitions can be added under the data table by hour, day, or week. The time granularity can be set and adjusted according to actual needs, such as according to the service level agreement (SLA) between the analysis platform and the third-party manufacturer.

[0058] Optionally, in some embodiments of the present application, the data table management device 11 may be disposed in the ETL tool 13, or may be a software framework independent of the ETL tool 13. The data table management device 11 and the ETL tool 13 may be stored in the second storage area 110, or may be stored in other storage areas on the second server 210.

[0059] Optionally, in some embodiments of the present application, the first storage area 100 and the second storage area 110 may not be storage areas on the server, but storage areas of other cloud storage platforms, distributed storage systems or computer devices (such as desktop computers, industrial computers) with data storage functions.

[0060] Combine the following Figure 3 A schematic diagram of data interaction between a first server 200 and a second server 210 provided in an embodiment of the present application is exemplarily introduced.

[0061] 30. Setting time information of table partitions in the analysis database 12 according to the data analysis cycle of the analysis service provided by the second server 210, and setting partition sizes of table partitions in the analysis database 12 according to the hardware information of the second server 210;

[0062] In some embodiments of the present application, the second storage area 110 is provided with an analysis database 12, and the analysis database 12 includes a plurality of data tables, and each data table can be divided into a plurality of table partitions according to actual needs. Each table partition has attributes such as time information and partition size information. The time information of the table partition includes a start time and an end time, and the time information of the table partition can be set according to the SLA of the second server 210. The content of the SLA can include the data analysis cycle of the analysis service provided, the data query cycle of the query service, etc. For example, if the data analysis cycle of the analysis service provided by the second server 210 is a natural day, the start time of the table partition can be set to the time when a natural day starts, and the end time of the table partition can be set to the time when a natural day ends.

[0063] Furthermore, the partition size of the appropriate table partition can also be set according to the disk I / O performance, memory performance and processor performance of the second server 210. For example, the rule of the partition size of the table partition can be: the base size is S, the jitter rate is m%, that is, the partition size of the table partition can be an arbitrary value from S*(1-m%) to S*(1+m%). Assume that the base size of the table partition is 500M and the jitter rate is 10% based on the disk I / O performance, memory performance and processor performance of the second server 210, that is, the table partition size can be an arbitrary value between 450M and 550M. Assuming that the analysis period provided by the second server 210 is a natural day, if the partition size allows, the data table is partitioned according to natural days as much as possible to store the source data of the first storage area 100.

[0064] 31. The ETL tool 13 extracts the data to be transferred from the first storage area 100.

[0065] In some embodiments of the present application, the data to be transferred is part of the source data currently extracted by the ETL tool 13 from the first storage area 100. The attribute information of the data to be transferred may include the data start time, the data end time and the data size. The amount of data to be transferred extracted by the ETL tool 13 each time must be smaller than the size of the table partition, and the time granularity of the data to be transferred must also be smaller than the time granularity of the table partition. For example, if the table partition size is 500M, the amount of data to be transferred extracted by the ETL tool 13 each time is less than 500M, such as 50M, 100M, 150M, etc., that is, the amount of data to be transferred extracted by the ETL tool 13 each time is 50M, or the amount of data to be transferred extracted by the ETL tool 13 each time is 100M, or the amount of data to be transferred extracted by the ETL tool 13 each time is 150M.

[0066] In some embodiments of the present application, the amount of data to be transferred extracted by the ETL tool 13 each time may also be different, but the amount of data to be transferred extracted by the ETL tool 13 each time must be smaller than the size of the table partition.

[0067] If the time granularity of the table partition is a natural day, the data to be transferred each time extracted by the ETL tool 13 must be data within the same natural day. In some embodiments of the present application, since the analysis database 12 contains multiple data tables. After the data to be transferred is extracted, it is also necessary to determine the data table corresponding to the data to be transferred and the time information of the most recent table partition of the data table. The most recent table partition of the data table is the most recently created table partition of the data table. The table structure of the data table can be shown in Table 1 below:

[0068] Table 1

[0069] Fields type Field Definition index string Database primary key table_name string Table name partition_id int ID of the table partition partition_starttime long Table partition start time partition_endtime long Table partition end time partition_size long Table partition size partition_current boolean Indicates whether it is the latest table partition

[0070] For example, the length of the index field of Table 1 is 36, the length of the table_name field is 1024, and the length of the partition_id field is 36.

[0071] In some embodiments, the analysis database 12 can establish multiple data tables in advance according to different business function requirements, and each data table is used to store data of a business function. The ETL tool 13 or the data table management device 11 can be used to determine which business function the data to be transferred belongs to, so as to determine the data table corresponding to the data to be transferred. When the data table corresponding to the data to be transferred is determined, the time information of the most recent table partition of the data table can be obtained through a database query statement. For example, the database query language is the Structured Query Language (SQL), and the following SQL statement can be used to query the specified data table and obtain the partition start time and partition end time of the most recent table partition of the data table: "SELECT * FROM Table_Partition WHERE table_name = '% table_name' and partition_current = true". The database query language is not limited to SQL, and can also be other data query languages.

[0072] For example, the table partition size is 500M, and the ETL tool 13 extracts 80M of data to be transferred from the first storage area 100 each time. The data table corresponding to the data to be transferred is the data table Tb1 in the analysis database 12, that is, the data to be transferred is transferred to the data table Tb1 of the analysis database 12 on the second storage area 110.

[0073] 32. The ETL tool 13 transmits the attribute information of the data to be transferred to the data table management device 11.

[0074] For example, the ETL tool 13 may transmit information such as the data start time, data end time, and data size of the data to be transferred to the data table management device 11 .

[0075] 33. The data table management device 11 determines whether it is necessary to create a new table partition for the data to be transferred.

[0076] In some embodiments, the source data of the first storage area 100 is transferred to the second storage area 110 in sequence according to the data generation time. The data table management device 11 can compare the start time and end time of the data to be transferred with the start time and end time of the most recent table partition of the data table to determine whether the data to be transferred falls within the time period of the most recent table partition, and then determine whether it is necessary to create a new table partition for the data to be transferred. For example, the start time of the data to be transferred is 2019-1-18, 08:00:00, and the end time is 2019-1-18, 15:00:00. The data table is partitioned according to natural days. The start time of the latest table partition of the data table is 2019-1-18, 00:00:00, and the end time is 2019-1-18, 23:59:59. It can be determined that the data to be transferred falls within the time period of the latest table partition; if the start time of the latest table partition of the data table is 2019-1-17, 00:00:00, and the end time is 2019-1-17, 23:59:59, it can be determined that the data to be transferred does not fall within the time period of the latest table partition.

[0077] If the data to be transferred does not fall within the time period of the most recent table partition, the data table management device 11 can directly determine that the data to be transferred does not meet the storage conditions for being stored in the most recent table partition, and a new table partition can be created based on the time information of the data to be transferred. For example, a new table partition can be created in the data table using the following SQL language:

[0078] UPDATE Table_Partiton SET partition_current=false WHERE partition_current=true and table_name='%table_name'

[0079] INSERT INTO Table_Partition VALUES("uuid",table_name,"uuid",$data_starttime,$data_endtime,$data_size,true);

[0080] After creating the table partition, you can also update the table partition information in the data table using the following SQL language:

[0081] UPDATE Table_Partition SET partition_size=partition_size+%data_sizeWHERE table_name='%table_name'AND partition_current=true.

[0082] When the data to be transferred falls within the time period of the most recent table partition, continue to determine whether the amount of data to be transferred is less than or equal to the remaining storage space of the most recent table partition. If the amount of data to be transferred is less than or equal to the remaining storage space of the most recent table partition, it indicates that the most recent table partition has enough space to store the data to be transferred. At this time, there is no need to create a new table partition, and the ETL tool 13 can store the data to be transferred in the most recent table partition. If the amount of data to be transferred is greater than the remaining storage space of the most recent table partition, it indicates that the most recent table partition does not have enough space to store the data to be transferred. At this time, a new table partition can be created based on the time information of the data to be transferred. For example, a new table partition can be created in the data table using the following SQL language:

[0083] UPDATE Table_Partiton SET partition_current=false AND partition_endtime=%data_starttime-1WHERE partition_current=true AND table_name='%table_name'

[0084] INSERT INTO Table_Partition VALUES("uuid",table_name,"uuid",$data_starttime,$data_endtime,$data_size,true);

[0085] After creating the table partition, you can also update the table partition information in the data table using the following SQL language:

[0086] UPDATE Table_Partition SET partition_size=partition_size+%data_sizeWHERE table_name='%table_name'AND partition_current=true.

[0087] 34. The data table management device 11 transmits the result of whether to create a table partition to the ETL tool 13. In some embodiments, the data table management device 11 transmits the result of whether to create a table partition to the ETL tool 13, so that when a new table partition is created for the data to be transferred, the ETL tool 13 stores the data to be transferred in the newly created table partition of the data table Tb1, and when no new table partition is created for the data to be transferred, the ETL tool 13 stores the data to be transferred in the nearest table partition of the data table Tb1.

[0088] 35. The ETL tool 13 stores the data to be transferred to the second storage area 110.

[0089] If the data table management device 11 creates a new table partition in the data table Tb1 for the data to be transferred, the ETL tool 13 stores the data to be transferred in the newly created table partition; if the data table management device 11 does not create a new table partition in the data table Tb1 for the data to be transferred, the ETL tool 13 stores the data to be transferred in the nearest table partition of the data table Tb1.

[0090] For example, Figure 4 As shown in the figure, in the data table Tb1, the size of the table partition is 500M, the data query granularity of the query service is one natural day, and the table partition is divided according to one natural day (24 hours). The start time of the data to be transferred d1 is 2019-1-18, 01:00:00, and the end time is 2019-1-18, 01:59:59. The data volume of the data to be transferred d1 is 100M. Table partition TP in the data table m For the recently created table partition, table partition TP m The start time is 2019-1-18, 00:00:00, the end time is 2019-1-18, 00:59:59, and the remaining storage space is 80M. By comparison, it can be determined that the time information of the data to be transferred d1 falls into the table partition TP m During the time period (table partition TP m The data to be transferred belongs to the same natural day as the data to be transferred d1), but the amount of data to be transferred d1 exceeds the table partition TP m The remaining storage space cannot store the data d1 to be transferred to the table partition TP. m , now create a new table partition TP m+1 Since the start time of the data to be transferred d1 is 2019-1-18, 01:00:00, and the end time is 2019-1-18, 01:59:59, the table partition TP m+1The start time is 2019-1-18, 01:00:00, the end time is 2019-1-18, 01:59:59, the storage space is 500M, due to the table partition TP m+1 It is a newly created table partition, and its remaining storage space is also 500M, and the table partition TP m+1 The data to be transferred belongs to the same natural day as the data d1 (that is, the time information of the data to be transferred d1 also falls into the table partition TP m+1 In this case, the ETL tool 13 can store the data d1 to be transferred to the table partition TP m+1 It can be understood that the table partition TP m TP with table partition m+1 Corresponding to the same natural day (2019-1-18).

[0091] For another example, the size of a table partition is 500M, the data query granularity of the query service is one natural day, and the table partition is divided according to one natural day (24 hours). The start time of the data to be transferred d1 is 2019-1-18, 01:00:00, and the end time is 2019-1-18, 01:59:59. The data volume of the data to be transferred d1 is 100M. Table partition TP in the data table m For the recently created table partition, table partition TP m The start time is 2019-1-18, 00:00:00, the end time is 2019-1-18, 00:59:59, and the remaining storage space is 120M. By comparison, it can be determined that the time information of the data to be transferred d1 falls into the table partition TP m The amount of data to be transferred d1 does not exceed the table partition TP m The remaining storage space, ETL tool 13 can store the data to be transferred d1 in table partition TP m , while the table partition TP m The time information will be changed to: start time is 2019-1-18, 00:00:00, end time is 2019-1-18, 01:59:59.

[0092] For another example, the size of a table partition is 500M, and the table partition is divided according to a natural day. The start time of the data to be transferred d1 is 2019-1-18, 08:00:00, and the end time is 2019-1-18, 15:00:00. The data volume of the data to be transferred d1 is 100M. Table partition TP in the data table m For the recently created table partition, table partition TP mThe start time is 2019-1-17, 00:00:00, the end time is 2019-1-17, 23:59:59, and the remaining storage space is 80M. By comparison, it can be determined that the time information of the data to be transferred d1 does not fall into the table partition TP m The data d1 to be transferred cannot be stored in the table partition TP during the time period m , now create a new table partition TP m+1 Since the data to be transferred d1 is from 2019-1-18, the table partition TP m+1 The start time is 2019-1-18, 00:00:00, the end time is 2019-1-18, 23:59:59, the storage space is 500M, due to the table partition TP m+1 It is a newly created table partition, and its remaining storage space is also 500M. The amount of data to be transferred d1 does not exceed the table partition TP m+1 The remaining storage space, at this time, the ETL tool 13 can store the data d1 to be transferred to the table partition TP m+1 .

[0093] In some embodiments of the present application, the message body M1 sent by the ETL tool 13 to the data table management device 11 is:

[0094]

[0095] The data table management device 11 responds to the message body M1 and can generate a message body M2:

[0096]

[0097] The data table management device 11 then determines whether it is necessary to create a new table partition in the data table based on the start time, end time and data volume of the data to be transferred, and the start time, end time and remaining storage space of the table partition, and returns the judgment result to the ETL tool 13.

[0098] In this way, the table partition is set to a reasonable partition size based on the SLA of the analysis service provided by the second server 210 and the hardware resources of the second server 210 itself. At the same time, for each extracted data to be transferred, it is determined whether to place it in a newly created table partition or an existing table partition, so that the data stored in the table partition is as close to the reasonable partition size as possible, which can minimize the system I / O waste and improve the stability of the analysis service.

[0099] Reference Figure 5As shown, a data storage method provided in an embodiment of the present application is applied to a second server 210. An analysis database 12 is provided on the second server 210, and the analysis database 12 includes multiple data tables, each of which can store different business function data. In this embodiment, the data storage method includes:

[0100] 500 , setting time information of table partitions of the data table based on the data analysis cycle of the analysis service provided by the second server 210 , and setting partition sizes of table partitions of the data table based on hardware information of the second server 210 .

[0101] 502 . Obtain attribute information of the data to be transferred extracted by the ETL tool 13 from the first storage area 100 .

[0102] 504 , determine the data table corresponding to the data to be transferred and the time information of the most recent table partition of the data table.

[0103] 506. Determine whether the data to be transferred meets the storage condition for being stored in the most recent table partition.

[0104] 508. If the data to be transferred meets the storage condition for being stored in the most recent table partition, the data to be transferred is stored in the most recent table partition.

[0105] 510. If the data to be transferred does not meet the storage condition for being stored in the most recent table partition, a new table partition is created according to the attribute information of the data to be transferred, and the data to be transferred is stored in the newly created table partition.

[0106] In some embodiments, the data stored in the analysis database 12 is generally time-sensitive. Data from half a year ago may be analyzed or queried on a monthly basis, data from three months ago may be analyzed or queried on a weekly basis, and data from the current month may be analyzed or queried on a daily basis. Dynamically adjusting the size of table partitions based on the time granularity of the services provided by the analysis database 12 and merging table partitions into table partitions that are more suitable for analysis or query services can reduce the hardware resource occupancy rate when responding to analysis or query services and shorten the response time of analysis or query services.

[0107] Combine the following Figure 6 The following is a schematic diagram of data interaction between the data table management device 11 and the analysis database 12 provided in the embodiment of the present application. The analysis database 12 has a partition merging function.

[0108] 60 . The data table management device 11 obtains time information of a table partition of a designated data table in the analysis database 12 .

[0109] In some embodiments of the present application, the designated data table is a data table that needs to be partitioned and merged in the analysis database 12, and the designated data table can be determined according to the actual merging requirements. The time information of the table partition includes the start time and the end time.

[0110] 61. The data table management device 11 determines the table partitions that can be merged in the specified data table according to the preset partition merging rule.

[0111] In some embodiments of the present application, the data table management device 11 can dynamically merge table partitions according to the time granularity of the analysis or query service, so as to optimize the disk I / O performance, processor occupancy rate, etc. when responding to the analysis or query service. The preset partition merging rules can be set according to actual needs to make the table partitions as close to a reasonable size as possible through partition merging. For example, the preset partition merging rules can be set in combination with business scenarios. For data with a long storage time, the time granularity of the analysis or query service provided is often coarser, such as data stored more than three months ago, and the query time granularity is based on the month dimension. For data with a newer storage time, the time granularity of the analysis or query service provided is often finer, such as data stored within one month, and the query time granularity is based on the day dimension.

[0112] The parameters that can be defined by the preset partition merge rule include: the ID of the data table to be merged, the time span of the table partitions to be merged, the size of the merged new table partition, and the number of table partitions to be merged each time. For example, the partition rule of the data table Tb1 from 2018-1-1 to 2018-1-31 is to partition according to natural days, the table partition size is a value between 450M and 550M, the preset partition merge rule is to merge the table partitions from 2018-1-1 to 2018-1-31 in the data table Tb1, the partition threshold of the new table partition is 2G, and two adjacent table partitions can be merged each time. Based on the above defined parameters, the table partitions from 2018-1-1 to 2018-1-31 can be scanned from the data table Tb1 according to the preset partition merge rule. For example, the table partitions that meet the preset partition merging rules can be scanned in the data table Tb1 by the following SQL language: SELECT * FROM Table_Partition WHERE table_name = '% table_name' ORDER BY partition_starttime ASC.

[0113] 62. The data table management device 11 sends the partition merging result of the designated data table to the analysis database 12.

[0114] After the data table management device 11 determines the table partitions that can be merged in the specified data table in the analysis database 12, it continues to determine the merging results of these table partitions. For example, multiple table partitions that meet the preset partition merging rules are obtained by scanning from the data table Tb1, including table partition TP1, table partition TP2, table partition TP3, table partition TP4, ..., table partition TP n . Table partition TP1, table partition TP2, table partition TP3, ..., table partition TP n . Table partition TP1~TP n The chronological order may be selected. The following description takes the example of trying to merge two table partitions each time to determine the merge result of the table partitions, but it is not limited to two table partitions, and you can also try to merge three adjacent table partitions each time, or merge more than three table partitions.

[0115] When merging partitions, they are also merged in the order of the table partitions, and it is determined whether the sum of the partition sizes of table partition TP1 and table partition TP2 is greater than the partition threshold of the new table partition. The partition size of table partition TP1 and table partition TP2 can be added and compared with the partition threshold of the new table partition to determine whether the sum of the partition sizes of table partition TP1 and table partition TP2 is greater than the partition threshold of the new table partition. If the sum of the partition sizes of table partition TP1 and table partition TP2 is less than or equal to the partition threshold of the new table partition, table partition TP1 and table partition TP2 are merged to obtain the first merged table partition TP1. 11 For example, you can remove the partition size restriction of table partition TP1, write the content of table partition TP2 into table partition TP1, and name table partition TP1 as the first merged table partition TP2. 11 , and delete table partition TP2; you can also create a new table partition and name it the first merged table partition TP 11 , write the contents of table partitions TP1 and TP2 into the first merged table partition TP 11 , and delete table partition TP1 and table partition TP2.

[0116] In some embodiments, if the sum of the partition sizes of table partition TP1 and table partition TP2 is less than or equal to the partition threshold of the new table partition, it indicates that table partition TP1 and table partition TP2 can be merged. In this case, table partition TP1 and table partition TP2 can be merged to obtain a first merged table partition TP1. 11 For example, you can merge table partition TP1 and table partition TP2 using the following SQL language:

[0117] UPDATE Table_Partition SET partition_endtime=%TP2_partition_endtimeWHERE table_name='%table_name'AND partition_id='%TP1_partition_id'

[0118] DELETE FROM Table_Partition WHERE partition_id='%TP2_partition_id'.

[0119] After the table partition TP1 and the table partition TP2 are merged, continue to determine the first merged table partition TP 11 Is the sum of the partition sizes of the first merged table partition TP and table partition TP3 greater than the partition threshold of the new table partition? 11 The sum of the partition sizes of table partition TP3 is less than or equal to the partition threshold of the new table partition, and table partition TP3 is merged into the first merged table partition TP 11 If the first merge table partition TP 11 The sum of the partition sizes of table partition TP3 and table partition TP4 is greater than the partition threshold of the new table partition. Continue to determine whether the sum of the partition sizes of table partition TP3 and table partition TP4 is greater than the partition threshold of the new table partition until the table partition TP4 is n Perform partition merging judgment.

[0120] If the sum of the partition sizes of table partition TP1 and table partition TP2 is greater than the partition threshold of the new table partition, continue to determine whether the sum of the partition sizes of table partition TP2 and table partition TP3 is greater than the partition threshold of the new table partition until the table partition TP n Perform partition merging judgment.

[0121] When the table partitions TP1 to TP n After determining the partition merging, the data table management device 11 may send the partition merging result to the analysis database 12.

[0122] 63. The analysis database 12 performs partition merging on the specified data table based on the partition merging result. For example, the partition merging result received by the analysis database 12 includes: [table partition TP1 and table partition TP2 are merged to obtain the first merged table partition TP 11 , table partition TP3, table partition TP4, and table partition TP5 are merged to obtain the second merged table partition TP 12 , table partition TP6]. The analysis database 12 merges the table partition TP1 and the table partition TP2 in the specified data table to obtain a new table partition TP 11 , the new table partition ID is TP11 The analysis database 12 also merges the table partitions TP3, TP4 and TP5 in the specified data table to obtain another new table partition TP 12 , the new table partition ID is TP 12 .

[0123] In some embodiments, taking HTTP Restful service as an example, the requested URL address is: http: / / pm_ip:pm_port / partitions / merge, and the data table management device 11 sends a message body M3 to the analysis database 12:

[0124]

[0125]

[0126] If the analysis database 12 returns a status code 200 for the message body M3, it indicates that the first table partition and the second table partition are merged successfully.

[0127] like Figure 7 As shown, assuming that the partition threshold TH of the new table partition is 800M, multiple table partitions that meet the preset partition merging rules are scanned from the data table Tb1, including: table partition TP1, partition size is 300M; table partition TP2, partition size is 400M; table partition TP3, partition size is 300M; table partition TP4, partition size is 200M; table partition TP5, partition size is 300M; table partition TP6, partition size is 400M. First, take table partition TP1 and table partition TP2. Since 300M+400M<800M, it means that table partition TP1 and table partition TP2 can be merged. Merge table partition TP1 and table partition TP2 to obtain the first merged table partition TP 11 , its partition size is 700M; then take table partition TP3, because 700M+300M>800M, indicating that the first merged table partition TP 11 The table partition TP3 cannot be merged any further, so the first merged table partition TP is abandoned. 11 Merge with table partition TP3; then take table partition TP4. Since 300M+200M<800M, it indicates that table partition TP3 and table partition TP4 can be merged. Merge table partition TP3 and table partition TP4 to obtain the second merged table partition TP 12 , its partition size is 500M; then take table partition TP5, since 500M+300M=800M, it indicates that the second merged table partition TP 12 You can also continue to merge with table partition TP5, merging table partition TP5 into the second merged table partition TP 12 , at this time the second merge table partition TP12 The partition size changes from 500M to 800M; then take table partition TP6. Since 800M+400M>800M, it indicates that the second merged table partition TP 12 The table partition TP6 cannot be merged any further, so the second merged table partition TP is abandoned. 12 Merge with table partition TP6; the final partition merging result is: [Table partition TP1 and table partition TP2 are merged to obtain the first merged table partition TP 11 , table partition TP3, table partition TP4, and table partition TP5 are merged to obtain the second merged table partition TP 12 , table partition TP6].

[0128] Reference Figure 8 As shown, a data storage method provided in an embodiment of the present application is applied to the second server 210. In this embodiment, the data storage method includes:

[0129] 800. Scan the specified data table in the analysis database 12 to obtain multiple table partitions TP1-TP2 that meet the preset partition merging rules. n , the multiple table partitions TP1~TP n Sort by chronological order.

[0130] 802. Determine whether the sum of the partition sizes of the adjacent table partitions TP1 and TP2 is greater than the partition threshold of the new table partition.

[0131] 804. If the sum of the partition sizes of table partition TP1 and table partition TP2 is less than or equal to the partition threshold of the new table partition, merge table partition TP1 and table partition TP2 to obtain a first merged table partition TP 11 .

[0132] 806. Determine the first merge table partition TP 11 Whether the sum of the partition sizes of table partition TP3 is greater than the partition threshold of the new table partition.

[0133] 808. If the first merge table partition TP 11 The sum of the partition sizes of table partition TP3 is less than or equal to the partition threshold of the new table partition, and table partition TP3 is merged into the first merged table partition TP 11 .

[0134] 810. If the first merge table partition TP 11 The sum of the partition sizes of table partition TP3 and table partition TP4 is greater than the partition threshold of the new table partition. Continue to determine whether the sum of the partition sizes of the adjacent table partition TP3 and table partition TP4 is greater than the partition threshold of the new table partition until the table partition TP n Perform partition merging judgment.

[0135] 812. If the sum of the partition sizes of table partition TP1 and table partition TP2 is greater than the partition threshold of the new table partition, continue to determine whether the sum of the partition sizes of the adjacent table partition TP2 and table partition TP3 is greater than the partition threshold of the new table partition, until the table partition TP n Perform partition merging judgment.

[0136] 814. Output multiple table partitions TP1~TP n The partition merging result of the partition merging is stored in the analysis database 12 .

[0137] refer to Fig. 9 , is a schematic diagram of the hardware structure of the second server 210 provided in the embodiment of the present application. Fig. 9 As shown, the second server 210 may include a processor 2001, a memory 2002, and a communication bus 2003. The memory 2002 is used to store one or more computer programs 2004. The one or more computer programs 2004 are configured to be executed by the processor 2001. The one or more computer programs 2004 include instructions, and the above instructions can be used to be executed to implement the data storage method in the second server 210.

[0138] It is understandable that the structure shown in this embodiment does not constitute a specific limitation on the second server 210. In other embodiments, the second server 210 may include more or fewer components than shown, or combine or split some components, or arrange the components differently.

[0139] The processor 2001 may include one or more processing units, for example, the processor 2001 may include an application processor (AP), a modem, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0140] The processor 2001 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 1001 is a cache memory. The memory may store instructions or data that the processor 2001 has just used or circulated. If the processor 2001 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 2001, and thus improves the efficiency of the system.

[0141] In some embodiments, the processor 2001 may include one or more interfaces. The interface may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a SIM interface, and / or a USB interface, etc.

[0142] In some embodiments, the memory 2002 may include a high-speed random access memory, and may also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices. This embodiment also provides a computer storage medium, in which a computer instruction is stored. When the computer instruction is executed on an electronic device, the electronic device executes the above-mentioned related method steps to implement the data storage method in the above-mentioned embodiment.

[0143] This embodiment also provides a chip, which is electrically connected to an electronic device and controls the electronic device to execute the above-mentioned related method steps to implement the data storage method in the above-mentioned embodiment. This embodiment also provides a computer program product, which, when running on a computer, enables the computer to execute the above-mentioned related steps to implement the data storage method in the above-mentioned embodiment.

[0144] In addition, an embodiment of the present application also provides a device, which may specifically be a chip, component or module, and the device may include a connected processor and memory; wherein the memory is used to store computer-executable instructions, and when the device is running, the processor may execute the computer-executable instructions stored in the memory so that the chip executes the data storage method in the above-mentioned method embodiments.

[0145] Among them, the second server, computer storage medium, computer program product or chip provided in this embodiment are all used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding methods provided above, and will not be repeated here.

[0146] Through the description of the above implementation methods, technical personnel in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0147] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0148] The unit described as a separate component may or may not be physically separated, and the component shown as a unit may be one physical unit or multiple physical units, that is, it may be located in one place or distributed in multiple different places. Some or all of the units may be selected according to actual needs to achieve the purpose of the embodiment.

[0149] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0150] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium, including several instructions to enable a device (which can be a single-chip microcomputer, chip, etc.) or a processor (processor) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), disk or optical disk and other media that can store program code.

[0151] The above description is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions within the technical scope disclosed in the present application should be included in the protection scope of the present application.

Claims

1. A data storage method, applied to a database, It is characterized in that The database includes at least one data table, and the data storage method includes: In response to storage requirements of the data to be transferred, determining a data table in the database corresponding to the data to be transferred; Obtaining time information and remaining storage space of the most recent table partition of the data table, and time information and data volume of the data to be transferred; If the time information of the most recent table partition does not match the time information of the data to be transferred and / or the remaining storage space of the most recent table partition is less than the data volume of the data to be transferred, a new table partition is created in the data table, and the data to be transferred is stored in the newly created table partition; Screening table partitions in the data table that can be merged according to a preset partition merging rule, and merging the screened table partitions, wherein the preset partition merging rule defines merging time information and a partition merging threshold corresponding to the merging time information; The step of screening the table partitions in the data table that can be screened according to the preset partition merging rule, and merging the screened table partitions includes: Filter out n table partitions whose time information falls within the merged time information from the data table, wherein the n table partitions are arranged in sequence according to time order; Sequentially attempt to merge m adjacent table partitions starting with each of the table partitions, where m and n are both positive integers, and m is less than n; If the sum of the partition sizes of m adjacent table partitions is less than or equal to the partition threshold, the m adjacent table partitions are merged into a new table partition, and partition merging attempts are continued for the m adjacent table partitions starting with the new table partition; If the sum of the partition sizes of m adjacent table partitions is greater than the partition threshold, partition merging of the m adjacent table partitions is abandoned.

2. The data storage method according to claim 1, It is characterized in that The data to be transferred includes data attributes, each data table in the database includes table attributes, and the table attributes are used to indicate data attributes of data that can be stored in the data table. The determining of the data table in the database corresponding to the data to be transferred includes: The data table corresponding to the data to be transferred is determined according to the data attributes of the data to be transferred and the table attributes of each of the data tables in the database.

3. The data storage method according to claim 2, It is characterized in that The data to be transferred is stored in a first server, and the database is deployed in a second server. The method further includes: The data to be transferred is extracted from the first server using a preset data extraction tool, and data attributes of the data to be transferred are sent to the second server.

4. The data storage method according to claim 3, It is characterized in that The method further comprises: Setting the time granularity of the table partition of the data table according to the service level agreement (SLA) of the second server; The partition size of the table partition of the data table is set according to the hardware configuration information of the second server, wherein the hardware configuration information at least includes disk read / write (I / O) performance.

5. The data storage method according to claim 4, It is characterized in that The method comprises: If the time information of the most recent table partition and the time information of the data to be transferred do not belong to the same time granularity, it is determined that the time information of the most recent table partition does not match the time information of the data to be transferred.

6. The data storage method according to claim 4, It is characterized in that The time information of the data to be transferred includes a start time and an end time. The amount of data to be transferred extracted by the preset data extraction tool from the first server each time is smaller than the partition size of the table partition, and the start time and end time of the data to be transferred belong to the same time granularity.

7. The data storage method according to claim 1, It is characterized in that The method further comprises: If the time information of the most recent table partition matches the time information of the data to be transferred and the remaining storage space of the most recent table partition is greater than or equal to the data volume of the data to be transferred, the data to be transferred is stored in the most recent table partition.

8. The data storage method according to claim 7, It is characterized in that The time information of the data to be transferred and the most recent table partition both include a start time and an end time. After storing the data to be transferred in the most recent table partition, the method further includes: If the end time of the data to be transferred is greater than the end time of the most recent table partition, the end time of the most recent table partition is updated to the end time of the data to be transferred.

9. The data storage method according to claim 1, It is characterized in that The obtaining time information of the most recent table partition of the data table includes: The start time and end time of the most recent table partition of the data table are obtained using a preset database query statement.

10. The data storage method according to claim 1, It is characterized in that Create a new table partition in the data table, including: The new table partition is created in the data table based on the time information of the data to be transferred.

11. The data storage method according to claim 1, It is characterized in that The database is deployed on a second server, and the method further includes: The merging time information is set according to the SLA of the second server.

12. A computer-readable storage medium, It is characterized in that The computer-readable storage medium stores computer instructions, and when the computer instructions are executed on an electronic device, the electronic device executes the data storage method according to any one of claims 1 to 11.

13. An electronic device, It is characterized in that The electronic device comprises a processor and a memory, wherein the memory is used to store instructions, and the processor is used to call the instructions in the memory, so that the electronic device executes the data storage method described in any one of claims 1 to 11.

Citation Information

Patent Citations

  • Data acquisition system storage management method

    CN101216821A

  • Method and device for data storage

    CN105095393A