Data table dividing method and device, electronic equipment and computer storage medium

Abstract

The invention is suitable for the technical field of database table division, and provides a table division method and device of a data table, electronic equipment and a computer storage medium, the method comprises the following steps: monitoring a first data total amount of a first data table, the first data table being a data table of which data storage is not finished at present; under the condition that the total amount of the first data is larger than a preset threshold value, a second data table corresponding to the first data table is created, and the structure of the second data table is the same as that of the first data table; according to the preset duration and the creation time of the second data table, determining storage starting time corresponding to the second data table; and storing the data in the second data table according to the storage starting time. The data tables are divided according to the data volume, so that the data in each data table is balanced, empty tables are effectively reduced, the resource utilization efficiency is improved, the situation of excessive data in the data tables is avoided, and the data query efficiency in a big data scene is improved.

Description

Technical Field

[0001] The present application belongs to the technical field of database table partitioning, and in particular, relates to a method, device, electronic device and computer storage medium for partitioning a data table. Background Art

[0002] In today's digital age, data collection and analysis of various devices based on data monitoring platforms has become an indispensable data processing method for various industries. However, data monitoring often requires the collection or reception of a large amount of data. If all data is stored in a data table, the database table will be very large, affecting the data query speed. Therefore, it is necessary to split the data table to improve the data query speed.

[0003] When faced with scenarios with large amounts of data, the currently commonly used solution is to partition tables by time range, splitting a data table into multiple data tables. However, this method will result in uneven distribution of data in the data tables. There will still be data tables with too much data, affecting the overall query efficiency of the data. In addition, it is difficult to balance the time range of the partitioned tables, which will generate a large number of empty tables and occupy system resources.

[0004] Therefore, how to improve resource utilization and ensure data query efficiency while splitting data tables has become a technical problem that continues to be solved. Summary of the invention

[0005] The embodiments of the present application provide a method, device, electronic device and computer storage medium for splitting a data table, which can solve the problem of how to improve resource utilization and ensure data query efficiency while splitting a data table.

[0006] In a first aspect, an embodiment of the present application provides a method for dividing a data table, including:

[0007] Monitoring the total amount of first data in a first data table, where the first data table is a data table in which data storage has not yet been completed;

[0008] When the total amount of the first data is greater than a preset threshold, a second data table corresponding to the first data table is created, and the structure of the second data table is the same as that of the first data table;

[0009] Determine the storage start time corresponding to the second data table according to the preset duration and the creation time of the second data table;

[0010] The data is stored in the second data table according to the storage start time.

[0011] In some embodiments, monitoring the first data amount of the first data table includes:

[0012] Obtaining the attribute information corresponding to each data table in the database, wherein the attribute information corresponding to each data table includes the storage end time and identification information corresponding to the data table;

[0013] Filter out first attribute information whose storage end time is a null value from the attribute information corresponding to each data table, where the first attribute information corresponds to the first data table;

[0014] According to the first identification information in the first attribute information, the total amount of first data in the first data table is queried.

[0015] In some embodiments, the attribute information corresponding to each data table also includes the total amount of data corresponding to the data table, and the method further includes:

[0016] Based on a preset time interval, third attribute information satisfying a first condition is selected from the attribute information corresponding to each data table, where the first condition is that the storage end time in the attribute information is not a null value and the total amount of data is a null value;

[0017] Determine the sum of the storage end time in the third attribute information and the preset duration;

[0018] When the sum result is less than or equal to the current time, query the third data table corresponding to the third attribute information according to the identification information in the third attribute information;

[0019] The total amount of data in the third data table is counted, and the total amount of data in the third attribute information is updated.

[0020] In some embodiments, the attribute information corresponding to each data table also includes the storage start time and storage end time corresponding to the data table. According to the storage start time, before storing the data in the second data table, it also includes:

[0021] Determine the storage end time corresponding to the first data table according to the storage start time, and update the first attribute information;

[0022] Determine the second identification information corresponding to the second data table according to the first identification information and the storage start time;

[0023] According to the second identification information, second attribute information corresponding to the second data table is generated and stored.

[0024] In some embodiments, the method further comprises:

[0025] When the target data is received, the target generation time and target data type of the target data, as well as the data type corresponding to each data table in the database are obtained;

[0026] Match the target data type with the corresponding data type of each data table;

[0027] In the case that the target data type does not match the data type corresponding to each data table, a fourth data table is created in the database, and the target data is stored in the fourth data table.

[0028] In some embodiments, the database includes a fifth data table, and the storage end time corresponding to the fifth data table is earlier than the storage start time corresponding to the second data table. The method further includes:

[0029] When the target data type matches the data type corresponding to the fifth data table, obtaining a storage time period corresponding to the fifth data table, the storage time period consisting of a storage start time and a storage end time corresponding to the fifth data table;

[0030] When the target generation time is within the storage time period, the target data is stored in the fifth data table.

[0031] In a second aspect, an embodiment of the present application provides a data table sub-table device, the device comprising:

[0032] A monitoring module, used for monitoring the total amount of first data in a first data table, where the first data table is any data table in which data storage has not yet been completed;

[0033] A creation module, used for creating a second data table corresponding to the first data table when the total amount of the first data is greater than a preset threshold, wherein the structure of the second data table is the same as that of the first data table;

[0034] A determination module, used to determine the storage start time corresponding to the second data table according to a preset duration and the creation time of the second data table;

[0035] The storage module is used to store data in the second data table according to the storage start time.

[0036] In a third aspect, an embodiment of the present application provides an electronic device, comprising: a processor, a memory, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the electronic device implements a method for partitioning a data table as in any embodiment of the first aspect.

[0037] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, a method for partitioning a data table in any embodiment of the first aspect is implemented:

[0038] In a fifth aspect, an embodiment of the present application provides a computer program product. When the computer program product is run on an electronic device, the terminal device executes the method for splitting a data table in any embodiment of the first aspect above.

[0039] Compared with the prior art, the embodiments of the present invention have the following beneficial effects:

[0040] For the first data table that has not yet finished data storage, the total amount of the first data stored in it is monitored, and when the total amount of the first data is greater than a preset threshold, a sub-table corresponding to the first data table is created, that is, a second data table with the same structure as the first data table, and data is stored in the sub-table to continue the data storage action, thereby realizing dynamic sub-table operation based on data volume on the data table. In the face of big data scenarios, it is ensured that the total amount of data between different data tables after sub-table distribution is relatively balanced, avoiding the occurrence of empty tables and the situation where the amount of data in a single table is too large or too small, thereby reducing the total amount of indexes in the data table to improve the storage space utilization rate, and then improving the resource utilization rate. And when querying data, it is ensured that the query time of different data tables is similar, reducing the query of empty tables and small data tables, and ensuring the efficiency of data query. And by delaying the storage start time of the second data table by a preset time length, and setting the time when the first data ends data storage to be equal to the storage start time, ensuring that the second data table delays data storage, can reduce the probability of missing or erroneous storage of unstored data when data is sub-tabled, and improve the reliability of data table sub-table. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0042] Figure 1 It is a flowchart of a method for dividing a data table provided in an embodiment of the present application;

[0043] Figure 2 It is a workflow diagram of a data table partitioning method in an application scenario;

[0044] Figure 3 It is a flowchart of another method for dividing a data table provided in an embodiment of the present application;

[0045] Figure 4 This is a workflow diagram of a data table partitioning method in another application scenario;

[0046] Figure 5 It is a flowchart of another data table partitioning method provided in an embodiment of the present application;

[0047] Figure 6 It is a workflow diagram of a data table partitioning method in another application scenario;

[0048] Figure 7 is a schematic diagram of the structure of an electronic device provided by an embodiment of the present application;

[0049] Figure 8 It is a structural schematic diagram of a data table sub-table device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0050] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0051] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof.

[0052] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0053] As used in the specification and appended claims of this application, the term "if" can be interpreted as "when" or "uponce" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "uponce it is determined" or "in response to determining" or "uponce [described condition or event] is detected" or "in response to detecting [described condition or event]", depending on the context.

[0054] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0055] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0056] The way to divide a data table by time range is often to split the data table according to a certain time range, for example, split a data table into multiple data tables according to days or months. Or create a new table at a certain time interval to store new data, for example, create a new table every day to store the data of the day. However, since the way the data monitoring platform collects or receives data remains unchanged, the total amount of data generated by the equipment monitored by the data monitoring platform in different time periods still fluctuates, resulting in uneven distribution of data in the data table. In certain time periods, the amount of data is huge, and the amount of data in the data table is very large, making it difficult to increase the query rate. In time periods with sparse data, the amount of data in the data table is very small, and even an empty table appears, but when querying data, it is still necessary to query the data table with small data volume, which undoubtedly consumes performance and resources.

[0057] In addition, it is difficult to balance the time range of the table. If the time range is too small, a large number of empty tables will be generated, occupying system resources, and the time cost of locating table information during platform query will increase significantly; if the time range is too large, although the proportion of empty tables and the number of tables are reduced, the amount of data in a single table will increase dramatically, returning to the dilemma of too much data leading to slow queries. This method of sharding data tables according to time range cannot meet the user's requirements for reasonable use of system resources and fast data query in large data volume scenarios.

[0058] In response to the above problems, the embodiments of the present application provide a data table partitioning method, device, electronic device and computer storage medium. For data tables that have not yet completed data storage, partitioning is performed based on the amount of data in the data table, so as to achieve dynamic partitioning of data tables that have not yet completed data storage in large data volume scenarios, and also make the distribution of data in different data tables relatively balanced, avoid the occurrence of empty tables and the situation where the amount of data in a single table is too large or too small, reduce the total amount of data table indexes and queries on empty tables and tables with small data volumes, improve resource utilization efficiency and ensure data query efficiency. The storage start time of the partitioned table is postponed based on a preset time length, so that the partitioned table can store data only after a period of time has passed since its creation, which can reduce the probability of missing or erroneous storage of unstored data when partitioning data, and improve the reliability of the partitioned table.

[0059] The following describes the data table partitioning method of the present application through specific embodiments.

[0060] Figure 1 is a flow chart of a method for dividing a data table provided in an embodiment of the present application, such as Figure 1 The method shown comprises the following steps:

[0061] Step S101, monitoring the total amount of first data in a first data table.

[0062] The data table partitioning method provided in the embodiment of the present application can be applied to electronic devices such as servers, industrial control computers, desktop computers, laptop computers or embedded devices. The embodiment of the present application does not impose any restrictions on the specific type of electronic devices.

[0063] In this embodiment, the first data table is a data table in which data storage has not yet been completed.

[0064] Exemplarily, the database stores the business status field of each data table when storing data. The electronic device can query the business status field of each data table based on a specified time interval, for example, every 5 minutes (min) or every 10 minutes, etc., and determine the data table whose business status field is in progress as the data table that has not yet ended data storage, that is, the first data table, and count the total number of data in the first data table to determine the total amount of the first data. It can be understood that when there are multiple data tables with business status fields in progress, the electronic device determines multiple first data tables, and the electronic device will subsequently create a corresponding second data table for each first data table.

[0065] Alternatively, the database stores the data update time when each data table stores data. The electronic device can query the data update time of each data table based on a specified time interval, and calculate the time difference between the data update time and the current time. If the time difference is less than a preset value, for example, 30 seconds (s) or 5s, etc., it is determined that the data table where the data update timestamp is located is the first data table, and then the total amount of the first data is determined by counting the total number of data in the first data table.

[0066] Considering that it takes a long time to calculate the total amount of data in the statistical table each time, in order to improve the efficiency of table division, after determining the first data table, the electronic device can also obtain the first identification information corresponding to the first data table. For example, the first identification information can be the table name or serial number of the first data table, etc., which can uniquely indicate the first data table. The embodiment of the present application does not make specific restrictions. Based on the first identification information, the first total amount of data stored in the first data table is queried in the information record table of the database. The information record table includes attribute information corresponding to each data table in the database, and the attribute information includes the total amount of data corresponding to the data table.

[0067] In one implementation, monitoring the total amount of first data in the first data table includes:

[0068] Obtaining the attribute information corresponding to each data table in the database, wherein the attribute information corresponding to each data table includes the storage end time and the total amount of data corresponding to the data table;

[0069] Filter out first attribute information whose storage end time is a null value from the attribute information corresponding to each data table, where the first attribute information corresponds to the first data table;

[0070] According to the first identification information in the first attribute information, the total amount of first data in the first data table is queried.

[0071] The storage end time is used to indicate the moment when the data table ends data storage. The electronic device can obtain the attribute information corresponding to each data table in the database based on the specified time interval. And determine whether there is attribute information with a null value for the storage end time in the acquired attribute information. The attribute information with a null value for the storage end time is used as the first attribute information corresponding to the first data table, and the first data table is located in the database based on the (first) identification information in the first attribute information, and the total amount of data stored in the first data table is counted to obtain the first total amount of data.

[0072] When there is no attribute information with a storage end time of a null value in the attribute information corresponding to each data table, the electronic device ends the process and enters the step of monitoring the total amount of the first data in the first data table after a specified time interval.

[0073] It can be understood that when multiple attribute information with a null storage end time is screened out from the attribute information corresponding to each data table, the electronic device determines multiple first attribute information (i.e., multiple first data tables), and the electronic device executes steps S101 to S104 for each first data table in parallel, or traverses each first data table and executes steps S101 to S104 for each traversed first data table. The electronic device can query the information record table from the database to query the attribute information with a null storage end time, and store the queried attribute information in the local cache of the electronic device for subsequent traversal of each first data table.

[0074] In one application scenario, such as Figure 2 As shown, the electronic device obtains an information record table every 5 minutes, and queries all attribute information whose storage end time is null from the information record table. When it is determined that the number of filtered attribute information is equal to zero, it means that there is no first data table that has not yet ended data storage, and the process ends. If it is greater than zero, the queried attribute information is traversed from any filtered attribute information, and based on the data table name (an example of identification information) in the currently traversed attribute information, the total amount of data (an example of the total amount of first data) of the data table corresponding to the data table name (an example of first data) is queried.

[0075] In the above technical solution, the attribute information of the storage end time of the null value is determined as the attribute information corresponding to the first data table, and the first total amount of data stored in the first data table is queried based on the identification information in the attribute information, so as to perform subsequent operations. It is realized that the total amount of data is only queried for the data table whose storage end time is a null value, that is, the data storage has not yet ended, thereby avoiding the table sharding operation of all data tables and improving the table sharding efficiency. In addition, the first total amount of data can be obtained by simply querying and filtering the attribute information of the data table, avoiding the traversal of all data tables in the database and the statistics of the data amount, improving the data acquisition efficiency and thus improving the table sharding efficiency.

[0076] Step S102: when the total amount of the first data is greater than or equal to a preset threshold, create a second data table corresponding to the first data table.

[0077] In this embodiment, the structure of the second data table is the same as that of the first data table. The preset threshold value may be a value of one million or one and a half million. When the total amount of the first data is greater than or equal to the preset threshold value, the electronic device will create a sub-table in the database that is completely consistent with the structure of the first data table, namely, the second data table.

[0078] It should be noted that after adding the second data table to the database, the attribute information corresponding to the second data table is generated, thereby establishing an index of the second data table in the database so that the electronic device can subsequently query the table. The attribute information corresponding to the second data table is similar to the first attribute corresponding to the first data table, and may include identification information corresponding to the second data table and the total amount of data. It can be understood that before the total amount of data stored in the second data is counted, the total amount of data corresponding to the second data table is a null value. The identification information corresponding to the second data table can be obtained on the basis of the identification information corresponding to the first data table, for example, a timestamp is added after the identification information corresponding to the first data table, or a serial number timestamp is added before the identification information corresponding to the first data table. For example, the identification information corresponding to the first data table is system_data0, and the identification information corresponding to the second data table can be system_data1.

[0079] Step S103: determining the storage start time corresponding to the second data table according to the preset time length and the creation time of the second data table.

[0080] In this embodiment, the storage start time is equal to the time when the first data table ends data storage, that is, the storage start time corresponding to the second data table is equal to the storage end time corresponding to the first data table; the preset duration can be 5 minutes or 3 minutes, etc., which can be determined based on the query time of the first data table and the network delay time between the electronic device and the database. The embodiment of this application takes 5 minutes as an example. The electronic device will determine the sum of the creation time of the second data table and the preset duration as the storage start time corresponding to the second data table. For example, the time of creating the second data table is 10:00:00 on X-year-X-month-X-day, then the storage start time corresponding to the second data table is 10:05:00 on X-year-X-month-X-day.

[0081] The storage start time corresponding to the second data table is set after the creation time of the table. The data obtained after the storage start time can be subsequently stored in the second data table, giving the electronic device buffer time to store data in the first data table. If batch storage is encountered, the electronic device can also store data in the first data table, thereby reducing the probability of erroneously storing data that should be stored in the first data table in the second data table and omitting data from being stored in the first data table, improving the reliability of data storage in the data partition table, and ensuring the accuracy of data query after partitioning.

[0082] In one implementation, the attribute information corresponding to each data table also includes the storage start time and storage end time corresponding to the data table. According to the storage start time, before storing the data in the second data table, it also includes:

[0083] Determine the storage end time corresponding to the first data table according to the storage start time, and update the first attribute information;

[0084] Determine the second identification information corresponding to the second data table according to the first identification information and the storage start time;

[0085] According to the second identification information, second attribute information corresponding to the second data table is generated and stored.

[0086] The electronic device may determine the storage start time corresponding to the second data table as the storage end time corresponding to the first data table, and replace the null value of the storage end time with the determined value in the first attribute information. The storage start time corresponding to the second data table is converted into a timestamp to obtain the timestamp corresponding to the second data table; the electronic device may replace the timestamp in the first identification information based on the timestamp corresponding to the second data table to obtain the second identification information; the first identification information is determined based on the data type and creation time corresponding to the first data table. The format of the identification information of the data table may be the data type corresponding to the data table_timestamp, for example, log_timestampX, which means a data table for storing log data created at the time indicated by timestamp X.

[0087] After the electronic device determines the second identification information, it will generate second attribute information and store the second attribute information in the information recording table. It can be understood that the electronic device can store the second attribute information in the information recording table in the local cache and the information recording table in the database respectively. The second attribute information may include storage start time, storage end time, second identification information, second data total amount and data type, wherein the storage start time is the time determined in step S103, the data type is the data type corresponding to the first data table, and the storage end time and the second data total amount can be set to null values ​​for subsequent updates.

[0088] Continue to refer Figure 2, for the currently traversed attribute information, based on the identification information in the attribute information, query the total amount of data in the data table corresponding to the attribute information (hereinafter referred to as the old data table), and when the total amount of data is greater than or equal to the preset threshold, create a data table (an example of the second data table) with a structure completely consistent with the data table, and insert a new record (an example of the second attribute information) into the information record table. The storage start time in the record is the creation time of the new data table plus 5 minutes. For example, the creation time of the new data table is 10:00:00 on November 5, 2024, plus 5 minutes to 10:05:00 on November 5, 2024, and 10:05:00 on November 5, 2024 is determined as the storage start time corresponding to the new data table. And convert the storage start time into the timestamp 1730772300000. If the table name of the old data table is tydata_1730298800000, the table name of the new data table in the record (an example of the second identification information) is tydata_1730772300000.

[0089] The purpose of adding 5 minutes is to prevent the situation where the wrong data table is inserted when there is batch insertion of data. For example, after a new data table is created, but the information record table has not been updated, the new data table is established. At this time, the acquired data should be inserted into the new table, but after querying the information record table, it is still indicated that the acquired data should be inserted into the old data table. It can be understood that the information record table can also include a data insertion mechanism corresponding to each data table in the database. Moreover, adding 5 minutes will not cause a sharp increase in the amount of data in the old data table, and will not bring too much burden to data query. At the same time, it can basically ensure that the new data table is created and the corresponding data in the information record table is updated.

[0090] The storage end time of the previous record (an example of the first attribute information) in the information record table is updated to the creation time of the new data table plus 5 minutes. Then the data is stored in the new data table, and it is determined whether the last attribute information is traversed. If so, the process ends. After 5 minutes, the information record table is read from the local cache, and all attribute information with a null storage end time is filtered out. If it is not the last attribute information, the next attribute information is traversed and a similar process is repeated.

[0091] Step S104, storing data in the second data table according to the storage start time.

[0092] When the second data table has not finished storing data, the electronic device can store data that meets specific conditions in the second data table. The specific conditions include that the generation time of the data is greater than or equal to the storage start time corresponding to the second data table, and the data type of the data matches the data type corresponding to the second data table.

[0093] For example, the storage start time is 09:00:00 on X-month-X-year, the data type corresponding to the second data table is voltage, the electronic device receives a voltage data at 12:00:00 on X-month-X-year, the data type corresponding to the voltage data is the same as the voltage corresponding to the second data table, and 12:00:00 on X-month-X-year is later than 09:00:00 on X-month-X-year. Moreover, the electronic device also obtains that the storage end time corresponding to the second data table is null (that is, the second data table has not yet ended data storage), and then stores the voltage data in the second data table.

[0094] In the embodiment of the present application, for the first data table that has not yet ended data storage, the total amount of the first data stored in the data is monitored, and when the total amount of the first data is greater than the preset threshold, a sub-table corresponding to the first data table is created, that is, a second data table with the same structure as the first data table, and data is stored in the sub-table to continue the data storage action, so as to realize the dynamic sub-table operation based on the data volume on the data table. In the face of big data scenarios, it is ensured that the distribution of the total amount of data between different data tables after sub-table is relatively balanced, and the situation of empty tables and single-table data volume being too large or too small is avoided, thereby reducing the total amount of indexes of the data table to improve the storage space utilization rate, and then improve the resource utilization rate. And when querying data, it is ensured that the query time of different data tables is similar, reducing the query of empty tables and small data tables, and ensuring the efficiency of data query. And by delaying the storage start time of the second data table by a preset time length, and setting the time when the first data ends data storage to be equal to the storage start time, it is ensured that the second data table delays the storage of data, which can reduce the probability of missing or mis-storing the unstored data when the data is sub-tabled, and improve the reliability of the data table sub-table.

[0095] Figure 3 is a schematic diagram of another data table partitioning method provided in an embodiment of the present application, such as Figure 3 The method shown comprises the following steps:

[0096] Step S201, when target data is received, obtain the target generation time and target data type of the target data, and the data type corresponding to each data table in the database.

[0097] The target data includes the target generation time and the target data type. After receiving the target data, the electronic device can obtain the target generation time and the target data type, and determine the data type corresponding to each data table in the database from the read information record table.

[0098] It can be understood that if the target data is received after the second data table is created, the database includes the second data table as well as the first data table.

[0099] In one implementation, before obtaining the target generation time and target data type of the target data, and the data type corresponding to each data table in the database, the method also includes: the electronic device performs a data preprocessing operation on the target data, and the data preprocessing operation includes at least one of data analysis, data cleaning, data screening or data normalization, so as to store the processed target data.

[0100] Step S202: Match the target data type with the data type corresponding to each data table.

[0101] The electronic device will search for the target data type from the data types corresponding to each data table. If the target data type is not found, it means that the target data type is received for the first time, and the process goes to step S203. If the target data type is found, the identification information corresponding to all data types matching the target data type is filtered out from the information recording table, and the data table indicated by the identification information is the data table that may store the target data. For each identification information filtered out, the storage end time and storage start time corresponding to the identification information are obtained from the information recording table. The storage end time and storage start time corresponding to the identification information constitute the storage time period corresponding to the identification information; compare whether the target generation time is within the storage time period corresponding to the identification information. If so, search the data table corresponding to the identification information in the database, and store the target data in the found data table.

[0102] Step S203: When the target data type does not match the data type corresponding to each data table, a fourth data table is created in the database, and the target data is stored in the fourth data table.

[0103] The electronic device will create a fourth data table in the database, generate attribute information corresponding to the fourth data table, add the attribute information to the information recording table, and store the target data in the fourth data table.

[0104] In the technical solution shown in step S201 to step S203, after receiving the data, the data type corresponding to the data is first matched with the data types corresponding to the existing data tables in the database. If the data type corresponding to the data does not match the data types corresponding to the existing data tables, a new data table is created to store the data. Each data table in the database stores data corresponding to its own data type, which facilitates subsequent query operations on the data.

[0105] In one implementation, the database includes a fifth data table, and the storage end time corresponding to the fifth data table is earlier than the storage start time corresponding to the second data table. The method further includes the following steps S204 to S205:

[0106] Step S204: when the target data type matches the data type corresponding to the fifth data table, obtain the storage time period corresponding to the fifth data table.

[0107] In this embodiment, the storage time period is composed of the storage start time and storage end time corresponding to the fifth data table. When the electronic device determines that the target data type matches the data type corresponding to the fifth data table, it can search the information recording table for the storage start time and storage end time corresponding to the fifth data table based on the identification information corresponding to the fifth data table, thereby obtaining the storage time period corresponding to the fifth data table.

[0108] Step S205: When the target generation time is within the storage time period, the target data is stored in a fifth data table.

[0109] In one application scenario, such as Figure 4 As shown, the electronic device obtains log data (an example of target data), performs data preprocessing operations on the log data, such as data parsing or data generalization, and obtains the data that needs to be stored in the database (that is, the processed data, including information such as data generation time and data type, an example of processed target data). The data type of the log data (an example of the target data type) is obtained from the processed data, and it is determined whether the attribute information corresponding to the data type is stored in the information record table (an example of matching the target data type with the data type corresponding to each data table). In order to improve the query efficiency, if the electronic device queries the information record table for the first time, it can query the relevant data from the database and store the query results in the local cache, and then query the information record table from the cache. For example, the data type of the log data is log_total, and the log data should be stored in the log total table. The data type corresponding to the log total table is log_total. The electronic device can query the attribute information of the data type log_total from the information record table and store the result in the local cache.

[0110] If the corresponding attribute information does not exist in the information record table (that is, the target data type does not match the data type corresponding to each data block), it means that the data table corresponding to the data type of the log data has not been created in the database. At this time, a corresponding data table is created according to the data type, and the attribute information of the data table is created in the information record table, in which the storage start time can be set to 0, the timestamp in the identification information can also be set to 0, and the storage end time is set to a null value. For example, if there is no attribute information corresponding to the data type log_total, a data table of the corresponding structure of the log_total type is created in the database, and the attribute information of the table is recorded in the information record table (the identification information is log_total_0, the storage start time is 0, the storage end time is null, the total amount of data is null, and the data type is log_total), and then the log data is inserted into the newly created data table.

[0111] If there is corresponding attribute information in the information record table, it means that there is a data table corresponding to the data type in the database, and all attribute table information corresponding to the data type in the information record table is obtained, for example, the attribute information corresponding to all data tables with data type log_total is obtained. Traverse each attribute information obtained, determine which attribute information's storage start time and storage end time the log data generation time (an example of the target generation time) is between (an example of the target generation time being within the storage time period corresponding to the fifth data table), and insert the log data into the data table corresponding to the attribute information. For example, the electronic device obtains 3 attribute information, where the storage start time to the storage end time are 0-10, 10-30, and 30-null respectively. If the log data generation time is 25, the log information is inserted into the data table corresponding to 10-30. If the log data generation time is 40, the log information is inserted into the data table corresponding to 30-null.

[0112] It can be understood that in the embodiments of the present application, data can be stored in the data table individually or in batches. When storing in batches, the generation time of the data to be stored can be divided, and the data can be stored in batches in the corresponding data table based on the divided generation time.

[0113] In an embodiment of the present application, when storing received data, the data type and generation time of the data are taken into account, and the data table in which the data should be stored is finally determined based on the dual factors. After determining that the data type of the data matches the data type corresponding to the fifth data table (that is, the data table in which storage has ended), if the generation time of the currently received data is within the storage time period of the data table in which storage has ended, the data will still be stored in the data table. The delayed reception of data is taken into account, so that the delayed received data can still be stored in the correct data table. After the data table is split, when there is a large network delay or equipment abnormality, the correct storage of the data can still be ensured, thereby improving the reliability of data storage after the data table is split and thus ensuring the accuracy of data query after the data table is split.

[0114] Figure 5 is a flow chart of another data table partitioning method provided in an embodiment of the present application, such as Figure 5 The method shown comprises the following steps:

[0115] Step S301: based on a preset time interval, third attribute information satisfying a first condition is screened out from the attribute information corresponding to each data table.

[0116] Among them, the attribute information corresponding to each data table also includes the total amount of data corresponding to the data table; the first condition is that the storage end time in the attribute information is not a null value, and the total amount of data is a null value. In other words, the attribute information that meets the first condition can indicate a data table that has ended data storage but the total amount of data is not yet clear. The preset time interval can be 1 hour (h) or 30 minutes, etc., and the embodiment of the present application takes 1h as an example. The electronic device can query at least one attribute information (that is, the third attribute information) that meets the first condition from the information recording table, and execute steps S302 to S304 for each third attribute information. It can be understood that if there is no third attribute information that meets the first condition in the attribute information corresponding to each data table, the process is terminated until step S301 is executed again after the preset time interval.

[0117] Step S302: determine the sum of the storage end time in the third attribute information and the preset duration.

[0118] Step S303: When the sum result is less than or equal to the current time, query the third data table corresponding to the third attribute information according to the identification information in the third attribute information.

[0119] The electronic device may search the database for a data table whose identification information is the identification information in the third attribute information, and obtain the third data table.

[0120] Step S304, counting the total amount of data in the third data table, and updating the total amount of data in the third attribute information.

[0121] The electronic device may count the total number of entries of the data stored in the third data table, use the total number of entries as the total amount of data, and update the total amount of data in the third attribute information to the counted total amount of data.

[0122] In one application scenario, such as Figure 6 As shown, taking the preset duration of 5 minutes as an example, the electronic device performs a data volume statistics timing task once every hour (an example of a preset time interval). After the timing task starts to execute, the electronic device first obtains all attribute information in the information record table whose storage end time is not a null value and whose total data volume is a null value. The electronic device traverses the queried attribute information to see if the number of attribute information is greater than 0. If it is equal to 0, it means that there is no need to perform a statistical operation on the total data volume, and it ends directly. Otherwise, for the currently traversed attribute information, it is determined whether the storage end time plus 5 minutes (an example of the sum result) is less than or equal to the current time (this can prevent data from being inserted into the data table, resulting in inaccurate statistics on the total data volume); if the current time is greater than the storage end time plus 5 minutes, the total data volume in the data table corresponding to the attribute information is queried, and the total data volume is updated to the attribute information corresponding to the information record table; otherwise, it is determined whether the currently traversed attribute information is the last one. If it is not the last one, the next attribute information is obtained, and it is determined whether the storage end time plus 5 minutes is less than or equal to the current time. Otherwise, this process ends.

[0123] In the embodiment of the present application, based on the specified time interval, specific attribute information is screened out to count the total amount of data in the data table corresponding to the data information, and the relevant part of the attribute information is updated. The accuracy of the total amount of data in the data table where data storage has ended can be ensured. Before calculating the total amount of data, the sum of the storage end time and the preset duration in the specific attribute information is determined, and the statistical action is triggered based on whether the sum is later than the current time, giving the data table corresponding to the specific attribute information a buffer time. Taking into account the data storage delay, the accuracy of the total amount of data statistics can be improved, so that the data table can be queried by data volume in the future.

[0124] In the scenario of large amounts of data, comparing the data table partitioning method in the embodiment of the present application (hereinafter referred to as the new partitioning method) with the traditional partitioning method by time range (taking one day as an example) (hereinafter referred to as the old partitioning method), the old partitioning method will result in one data table per day, and the new partitioning method will result in multiple data tables per day. Although the old partitioning method creates fewer data tables than the new partitioning method, when performing data queries, due to the excessive amount of data in one data table in the old partitioning method, the query time is likely to reach tens of seconds, several minutes or even longer, and such a response time is completely unbearable for users, and although the new partitioning method queries several more data tables, the query time of a single data table is not long, so it can respond to users faster, so the new partitioning method is still faster than the old partitioning method.

[0125] In addition, when the total amount of data in a period of time, the new table partitioning method can quickly calculate the total amount of data in this period based on the attribute information corresponding to each data table and the total amount of data corresponding to each data table during this period. The old table partitioning method needs to query all data tables in this period, count the amount of data in each data table, and then calculate the total amount of data, which seriously affects the query efficiency of the total amount of data.

[0126] Figure 7 This is a schematic diagram of the structure of an electronic device provided by an embodiment of the present application. Figure 7 As shown, the electronic device 6 of this embodiment includes: at least one processor 60 ( Figure 7 Only one is shown in the figure) a processor, a memory 61, and a computer program 62 stored in the memory 61 and executable on the at least one processor 60, and when the processor 60 executes the computer program 62, the steps in the above-mentioned method for dividing any data table are implemented.

[0127] The electronic device 6 may be a computing device such as a desktop computer, a notebook, a PDA, or a cloud server. The electronic device may include, but is not limited to, a processor 60 and a memory 61. Those skilled in the art will appreciate that Figure 7 It is only an example of the electronic device 6 and does not constitute a limitation on the electronic device 6. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, it may also include input and output devices, network access devices, etc.

[0128] The processor 60 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0129] In some embodiments, the memory 61 may be an internal storage unit of the electronic device 6, such as a hard disk or memory of the electronic device 6. In other embodiments, the memory 61 may also be an external storage device of the electronic device 6, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the electronic device 6. Further, the memory 61 may also include both an internal storage unit of the electronic device 6 and an external storage device. The memory 61 is used to store an operating system, an application program, a boot loader (BootLoader), data, and other programs, such as the program code of the computer program, etc. The memory 61 may also be used to temporarily store data that has been output or is to be output.

[0130] Corresponding to the data table partitioning method described in the above embodiment, Figure 8 The structure block diagram of the data table sub-table device provided in the embodiment of the present application is shown. For the convenience of explanation, only the part related to the embodiment of the present application is shown. Figure 8 , the device comprises:

[0131] A monitoring module 810 is used to monitor the total amount of first data in a first data table, where the first data table is any data table in which data storage has not yet been completed;

[0132] A creation module 820, configured to create a second data table corresponding to the first data table when the total amount of the first data is greater than a preset threshold, wherein the structure of the second data table is the same as that of the first data table;

[0133] A determination module 830, configured to determine a storage start time corresponding to the second data table according to a preset duration and a creation time of the second data table;

[0134] The storage module 840 is used to store data in the second data table according to the storage start time.

[0135] In some embodiments, the monitoring module is used to:

[0136] Obtaining the attribute information corresponding to each data table in the database, wherein the attribute information corresponding to each data table includes the storage end time and identification information corresponding to the data table;

[0137] Filter out first attribute information whose storage end time is a null value from the attribute information corresponding to each data table, where the first attribute information corresponds to the first data table;

[0138] According to the first identification information in the first attribute information, the total amount of first data in the first data table is queried.

[0139] In some embodiments, the attribute information corresponding to each data table also includes the total amount of data corresponding to the data table, and the device further includes:

[0140] A screening module, used to screen out third attribute information that meets a first condition from the attribute information corresponding to each data table based on a preset time interval, where the first condition is that the storage end time in the attribute information is not a null value and the total amount of data is a null value;

[0141] The determination module is further used to determine the sum of the storage end time in the third attribute information and the preset duration;

[0142] A query module, configured to query a third data table corresponding to the third attribute information according to the identification information in the third attribute information when the sum result is less than or equal to the current time;

[0143] The statistical module is used to count the total amount of data in the third data table, and the updating module is used to update the total amount of data in the third attribute information.

[0144] In some embodiments, the attribute information corresponding to each data table also includes a storage start time and a storage end time corresponding to the data table, and the device further includes:

[0145] The determination module is further used to determine the storage end time corresponding to the first data table according to the storage start time, and the update module is further used to update the first attribute information;

[0146] The determination module is further used to determine the second identification information corresponding to the second data table according to the first identification information and the storage start time;

[0147] The storage module is used to generate and store second attribute information corresponding to the second data table according to the second identification information.

[0148] In some embodiments, the apparatus further comprises:

[0149] An acquisition module is used to acquire the target generation time and target data type of the target data and the data type corresponding to each data table in the database when receiving the target data;

[0150] A matching module, used to match the target data type with the data type corresponding to each data table;

[0151] The creation module is used to create a fourth data table in the database and store the target data in the fourth data table when the target data type does not match the data type corresponding to each data table.

[0152] In some embodiments, the database includes a fifth data table, the storage end time corresponding to the fifth data table is earlier than the storage start time corresponding to the second data table, and the acquisition module is further used to acquire the storage time period corresponding to the fifth data table when the target data type matches the data type corresponding to the fifth data table, the storage time period consisting of the storage start time and the storage end time corresponding to the fifth data table;

[0153] The storage module is further configured to store the target data in a fifth data table when the target generation time is within the storage time period.

[0154] It should be noted that the information interaction, execution process, etc. between the above-mentioned devices / units are based on the same concept as the method embodiment of the present application. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0155] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0156] An embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the above-mentioned method embodiments can be implemented.

[0157] An embodiment of the present application provides a computer program product. When the computer program product runs on an electronic device, the electronic device can implement the steps in the above-mentioned method embodiments when executing the computer program product.

[0158] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium can at least include: any entity or device that can carry the computer program code to the camera / terminal device, recording medium, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, RandomAccess Memory), electric carrier signal, telecommunication signal and software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk or an optical disk. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electric carrier signals and telecommunication signals.

[0159] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0160] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0161] In the embodiments provided in the present application, it should be understood that the disclosed devices / network equipment and methods can be implemented in other ways. For example, the device / network equipment embodiments described above are merely schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0162] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0163] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A method for dividing a data table, characterized in that: include: Monitoring the total amount of first data in a first data table, where the first data table is a data table in which data storage has not yet been completed; When the first data amount is greater than a preset threshold, creating a second data table corresponding to the first data table, wherein the structure of the second data table is the same as that of the first data table; Determine the storage start time corresponding to the second data table according to the preset duration and the creation time of the second data table; Data is stored in the second data table according to the storage start time.

2. The method according to claim 1, characterized in that The monitoring of the first data amount of the first data table includes: Acquire the attribute information corresponding to each data table in the database, wherein the attribute information corresponding to each data table includes the storage end time and identification information corresponding to the data table; Filtering out first attribute information whose storage end time is a null value from the attribute information corresponding to each of the data tables, wherein the first attribute information corresponds to the first data table; The total amount of first data in the first data table is queried according to the first identification information in the first attribute information.

3. The method according to claim 2, characterized in that The attribute information corresponding to each of the data tables also includes the total amount of data corresponding to the data table, and the method further includes: Based on a preset time interval, third attribute information satisfying a first condition is screened out from the attribute information corresponding to each of the data tables, wherein the first condition is that the storage end time in the attribute information is not a null value and the total amount of data is a null value; Determine the sum of the storage end time in the third attribute information and the preset duration; When the sum result is less than or equal to the current time, querying a third data table corresponding to the third attribute information according to the identification information in the third attribute information; The total amount of data in the third data table is counted, and the total amount of data in the third attribute information is updated.

4. The method according to claim 2 or 3, characterized in that The attribute information corresponding to each of the data tables also includes a storage start time and a storage end time corresponding to the data table. Before storing data in the second data table according to the storage start time, the method further includes: Determine the storage end time corresponding to the first data table according to the storage start time, and update the first attribute information; Determine, according to the first identification information and the storage start time, second identification information corresponding to the second data table; According to the second identification information, second attribute information corresponding to the second data table is generated and stored.

5. The method according to claim 1, characterized in that The method further comprises: When receiving the target data, obtaining the target generation time and target data type of the target data, and the data type corresponding to each data table in the database; Matching the target data type with the data type corresponding to each of the data tables; In the case that the target data type does not match the data type corresponding to each of the data tables, a fourth data table is created in the database, and the target data is stored in the fourth data table.

6. The method according to claim 5, characterized in that The database includes a fifth data table, the storage end time corresponding to the fifth data table is earlier than the storage start time corresponding to the second data table, and the method further includes: In the case where the target data type matches the data type corresponding to the fifth data table, obtaining a storage time period corresponding to the fifth data table, wherein the storage time period is composed of a storage start time and a storage end time corresponding to the fifth data table; When the target generation time is within the storage time period, the target data is stored in the fifth data table.

7. A data table sub-table device, characterized in that: The device comprises: A monitoring module, used to monitor the total amount of first data in a first data table, where the first data table is any data table in which data storage has not yet been completed; a creation module, configured to create a second data table corresponding to the first data table when the total amount of the first data is greater than a preset threshold, wherein the structure of the second data table is the same as that of the first data table; A determination module, configured to determine a storage start time corresponding to the second data table according to a preset duration and a creation time of the second data table; A storage module is used to store data in the second data table according to the storage start time.

8. An electronic device, characterized in that: The electronic device comprises a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the electronic device implements the data table partitioning method as described in any one of claims 1 to 6.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method for dividing a data table according to any one of claims 1 to 6 is implemented.

10. A computer program product, characterized in that The invention comprises a computer program, and when the computer program is executed, the method for dividing a data table according to any one of claims 1 to 6 is executed.

Images