Table partitioning management method, apparatus, equipment, and storage medium based on index templates

Abstract

This application proposes a method, apparatus, device, and storage medium for table partitioning management based on index templates. The method includes: receiving a user's data insertion request, the data insertion request including the business data to be inserted and the name of the business table; determining an index template matching the business table name, the index template including modification records of partitioning units; and inserting the business data to be inserted into the target partitioned index table according to the business table name and the modification records of the partitioning units. This application records the modification records of the partitioning units in the metadata of the index template. Flexible management of partitioned index tables is achieved based on partitioning units, enabling CRUD operations on data in any partitioned index table. Through precise recording and management of modification records for partitioning units, CRUD operations on business tables are quickly converted into operations on certain specific partitioned index tables, reducing the number of index shards processed per operation and thus improving data processing efficiency.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and more specifically, to a table partitioning management method, apparatus, device, and storage medium based on an index template. Background Technology

[0002] The Elasticsearch search server (ES) currently supports rollover indexes, which can automatically partition tables based on index size, number of documents, and index duration. When specified conditions are met, a new index is automatically created, and a write alias is set to point to the new index. All subsequent updates will be written to the new index, and the old index becomes read-only.

[0003] However, when writing data, the data is written to the latest partition index table, while the older partition index table only provides data query functionality. Therefore, it is impossible to write data to a specific partition index table. Summary of the Invention

[0004] To address the aforementioned technical shortcomings, this application proposes a method, apparatus, device, and storage medium for partitioned table management based on index templates. This method manages partitioned index tables based on partitioning units, enabling data to be written to specified partitioned index tables based on business table names and partitioning units.

[0005] The first aspect of this application provides a table partitioning management method based on an index template, the method comprising:

[0006] Receive a user's data insertion request, the data insertion request including the business table name and the business data to be inserted;

[0007] Determine an index template that matches the business table name, wherein the index template includes modification records for the table sub-unit;

[0008] Based on the modification records of the business table name and the partition unit, the business data to be inserted is inserted into the target partition index table.

[0009] In some embodiments of this application, before determining the index template matching the business table name, the method further includes:

[0010] Receive a user's table creation request, which includes at least table partitioning unit and alias information;

[0011] Based on the table creation request, an index template is created, which includes the table partitioning unit and the alias information.

[0012] In some embodiments of this application, creating an index template according to the table creation request includes:

[0013] Create an index template;

[0014] Based on the alias information, generate the template matching pattern corresponding to the index template;

[0015] The alias information and the template matching pattern are stored in the index template;

[0016] Based on the partition unit and the current time, the modification record corresponding to the partition unit is stored in the metadata of the index template.

[0017] In some embodiments of this application, the method further includes:

[0018] Receive a table partitioning unit update request, the table partitioning unit update request including the business table name and the new table partitioning unit;

[0019] Based on the new table partitioning unit and the current timestamp, generate the modification record corresponding to the new table partitioning unit;

[0020] The modification records corresponding to the new table partitioning unit are stored in the metadata of the index template corresponding to the business table name.

[0021] In some embodiments of this application, determining the index template matching the business table name includes:

[0022] Obtain the alias information and template matching pattern corresponding to the first index template; the first index template is any index template that has been created so far.

[0023] Determine whether the alias information of the business table name and the first index template matches the template matching pattern corresponding to the first index template;

[0024] If so, the first index template is determined to be the index template that matches the business table name.

[0025] In some embodiments of this application, inserting the business data to be inserted into the target partition index table based on the modification record of the business table name and the partition unit includes:

[0026] Retrieve the modification records of the table partitioning unit from the metadata of the index template that matches the business table name;

[0027] Obtain the timestamp corresponding to the data insertion request;

[0028] Based on the modification records of the table sub-unit, determine the table sub-unit corresponding to the timestamp;

[0029] Generate a partition table name based on the business table name, the timestamp, and the partition unit corresponding to the timestamp;

[0030] The business data to be inserted is stored in the target partition index table corresponding to the partition table name.

[0031] In some embodiments of this application, the method further includes:

[0032] Receive a user's data query request, the data query request including the business table name, time parameter and data identifier of the data to be queried;

[0033] Based on the business table name, retrieve the modification records of the table partitioning unit from the metadata of the index template that matches the business table name;

[0034] Based on the modification records of the business table name, the time parameter, and the table sub-unit, generate the partition table name to be queried;

[0035] Based on the data identifier of the data to be queried, the data to be queried is retrieved from the partition index table corresponding to the partition table name to be queried.

[0036] In some embodiments of this application, the time parameter includes the start timestamp and end timestamp to be queried; generating the partition table name to be queried based on the business table name, the time parameter, and the modification records of the partition unit includes:

[0037] Retrieve the partition table names of all partition index tables from the index templates that match the business table names;

[0038] Obtain the modification time of all modification records from the modification records of the aforementioned table unit;

[0039] Sort the start timestamp, the end timestamp, and the modification times of all modified records in chronological order.

[0040] Based on the modification time of the modification records located between the start timestamp and the end timestamp after sorting, determine the partition table name of the partition index table for each time period between the start timestamp and the end timestamp from the partition table names of all partition index tables;

[0041] Based on the partition table names corresponding to each determined time period, generate a list of partition table names to be queried.

[0042] A second aspect of this application provides a table partitioning management device based on an index template, the device comprising:

[0043] The receiving module is used to receive a user's data insertion request, which includes a business table name and the business data to be inserted.

[0044] The determination module is used to determine the index template that matches the business table name, and the index template includes the modification records of the table sub-unit;

[0045] The data insertion module is used to insert the business data to be inserted into the target partition index table according to the business table name and the modification record of the partition unit.

[0046] A third aspect of this application provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method described in the first aspect above.

[0047] A fourth aspect of this application provides a computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the steps of the method described in the first aspect above.

[0048] The beneficial effects of this application are as follows:

[0049] This application's embodiment creates an index template when a user first requests table creation. The metadata of this index template records modifications to the table partitioning unit, making modifications to the partitioning unit convenient and quick. Based on the table partitioning unit, flexible management of the partitioned index table is achieved, enabling CRUD operations on data in any partitioned index table. Through precise recording and management of modifications to the table partitioning unit, CRUD operations on business tables are quickly converted into operations on specific partitioned index tables, reducing the number of index shards processed per operation and thus improving data processing efficiency. Attached Figure Description

[0050] The accompanying drawings, which form part of this specification, illustrate embodiments of this application and, together with the description, serve to explain the principles of this application.

[0051] This application can be more clearly understood with reference to the accompanying drawings and the following detailed description, wherein:

[0052] Figure 1 This illustration shows a schematic diagram of modifying the table partitioning unit based on an index template in an exemplary embodiment of this application;

[0053] Figure 2 A flowchart illustrating a table partitioning management method based on an index template provided in an exemplary embodiment of this application is shown.

[0054] Figure 3This illustration shows a schematic diagram of generating partition table names based on a single timestamp in an exemplary embodiment of this application;

[0055] Figure 4 This illustration shows a schematic diagram of generating a list of partition table names based on an initiation timestamp and an end timestamp in an exemplary embodiment of this application;

[0056] Figure 5 This illustration shows another flowchart of a table partitioning management method based on an index template provided in an exemplary embodiment of this application;

[0057] Figure 6 This illustration shows a schematic diagram of the structure of a table partitioning management device based on an index template provided in an exemplary embodiment of this application;

[0058] Figure 7 This invention provides a schematic diagram of the structure of an electronic device according to an exemplary embodiment of the present application.

[0059] Figure 8 A schematic diagram of a storage medium provided in an exemplary embodiment of this application is shown. Detailed Implementation

[0060] The embodiments of this application will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of this application. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concepts of this application. It will be apparent to those skilled in the art that this application can be implemented without one or more of these details. In other instances, some technical features well-known in the art have not been described to avoid confusion with this application.

[0061] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments according to this application. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms “comprising” and / or “including” are used in this specification, they indicate the presence of the stated features, integrals, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or combinations thereof.

[0062] Exemplary embodiments according to this application will now be described in more detail with reference to the accompanying drawings. However, these exemplary embodiments may be implemented in many different forms and should not be construed as being limited to the embodiments set forth herein. The drawings are not drawn to scale, and some details may be enlarged and omitted for clarity. The shapes of the various regions and layers shown in the figures, as well as their relative sizes and positional relationships, are merely exemplary and may deviate from reality due to manufacturing tolerances or technical limitations. Furthermore, those skilled in the art can design regions / layers with different shapes, sizes, and relative positions as needed.

[0063] The following is in conjunction with the instruction manual appendix. Figure 1-8 Several embodiments are given to describe exemplary implementations according to this application. It should be noted that the following application scenarios are shown only to facilitate understanding of the spirit and principles of this application, and the implementations of this application are not limited in any way. Rather, the implementations of this application can be applied to any applicable scenario.

[0064] This embodiment implements a table partitioning management method based on an index template. This method adds modification records of partitioning units to the metadata of the index template, enabling flexible management of partitioned index tables through partitioning units. When inserting data, the business data to be inserted is inserted into the target partitioned index table based on the modification records of the business table name and partitioning unit. This achieves writing data to a specified partitioned index table, solving the problem in related technologies that can only write data to the latest partitioned index table.

[0065] The table partitioning management method in this application is applied to the search server Elasticsearch (ES). ES is a distributed, highly scalable, and real-time search and data analysis engine. It can easily enable large amounts of data to be searched, analyzed, and explored. By fully utilizing the horizontal scalability of ES, data can become more valuable in production environments.

[0066] Before implementing the table partitioning management method based on index templates, it is first necessary to create an index template. This is done through the following steps S1-S2:

[0067] S1: Receive the user's table creation request, which includes at least the table partitioning unit and alias information.

[0068] The partitioning unit is the unit of division used to partition the index table based on the index template. The partitioning unit can be a time unit, a storage unit, an administrative division unit, etc. Specifically, partitioning by time unit can be by year, month, week, day, etc. Partitioning by storage unit can be by 100MB, 200MB, 1GB, etc. Partitioning by administrative division unit can be by country, province, district, city, etc.

[0069] Alias, also known as index aliases, are auxiliary names used to associate one or more existing partitioned index tables. Most Elasticsearch APIs (Application Programming Interfaces) accept index aliases to replace the real index names. This allows for transparent switching between underlying indexes without the user's awareness (without changing business code) and facilitates grouping and combining multiple indexes.

[0070] S2: Based on the table creation request, create an index template, which includes table partitioning unit and alias information.

[0071] When the Elasticsearch platform receives a user's table creation request, it does not create a partitioned index table to store the data. Instead, it first creates an index template. Specifically, the index template is created through the following steps S21-S24:

[0072] S21: Create an index template.

[0073] When the Elasticsearch platform receives a user's table creation request, it creates an index template. The index template defines settings and mappings that can be automatically applied when creating the index. If the name of the newly created partitioned index table matches the template matching pattern of an index template, the settings and mappings in the index template will be automatically applied to the newly created partitioned index table.

[0074] Index settings define index-level configurations, such as the number of shards and replicas. Index mappings define how documents and their fields are stored and indexed, such as field types and whether tokenization is used.

[0075] In Elasticsearch (ES), each document has associated metadata, such as its index (_index), mapping type (_type), and document ID (_id). Some of this metadata can be customized when creating index mappings. There are five types of index metadata: Identity meta-fields, Documentsource meta-fields, Indexing meta-fields, Routing meta-fields, and Other meta-fields. Other meta-fields can be used to store application-specific metadata (in JSON format). The content of the _meta field is completely unused by ES and is only stored in the template and index mappings, not in each document. The content of _meta supports querying and updating. This embodiment uses _meta to store modification records for each table partition.

[0076] S22: Generate a template matching pattern corresponding to the index template based on the alias information.

[0077] After creating the index template, a template matching pattern is generated based on the alias information carried in the table creation request. This template matching pattern includes the strings that must be included in the name of the index matching the template, and preset ellipsis characters. These preset ellipsis characters can be *, &, or $, etc. The preset ellipsis characters indicate that the name of the index matching the template can contain any other characters besides the specified required strings. In other words, if the index name contains the strings specified in the template matching pattern, then the index matches the template.

[0078] For example, assuming the alias information is data.table, the generated template matching pattern can be data.table.*. All subsequent partitioned index tables created that match this index template, such as data_table.202102, data_table.2021, etc., will automatically have the alias information data_table.

[0079] S23: Store alias information and template matching patterns in the index template.

[0080] The alias information carried in the table creation request and the template matching pattern generated based on the alias information are stored in the index template.

[0081] S24: Based on the partition unit and the current time, store the modification record corresponding to the partition unit in the metadata of the index template.

[0082] Use the current time as the timestamp for the first modification of the table partitioning unit of this index template. Based on the table partitioning unit and the timestamp carried in the table creation request, generate a modification record for the table partitioning unit and store the modification record in the metadata of this index template.

[0083] Specifically, modification records for each table partition are stored in Other meta-fields, which belong to one of the five metadata types. Other meta-fields can be used to store application-specific metadata (in JSON format).

[0084] For example, if the table partitioning unit is year, and the current time is October 5, 2018, then the generated modification record could be [timestamp: 20181005, partitioning unit: year]. This modification record is stored in the othermeta-field of the index template.

[0085] After creating the index template using the above method, the table partitioning unit can be modified at any time. The Elasticsearch platform receives the user's table partitioning unit update request, which includes the business table name and the new partitioning unit. Based on the new partitioning unit and the current timestamp, a modification record corresponding to the new partitioning unit is generated. This modification record corresponding to the new partitioning unit is stored in the metadata of the index template corresponding to the business table name.

[0086] For example, suppose the metadata of the index template records a modification record for the partition unit as [timestamp: 20181005, partition unit: year]. On September 5, 2019, a user's partition unit update request is received, which includes a new partition unit of month. The modification record corresponding to the new partition unit is then generated as [timestamp: 20190906, partition unit: month]. This modification record is appended to the mapping of the index template's metadata. That is, at this point, the metadata of the index template stores modification records including [timestamp: 20181005, partition unit: year] and [timestamp: 20190906, partition unit: month].

[0087] Modifying the partition unit will append a change record to the metadata of the corresponding index template. The record format is: {time string, partition unit}. Each update will take effect one day later (i.e., changes made on the current day take effect the day after), ensuring that it can be traced back to the partition index table where each piece of data is stored.

[0088] For example, if a table is created at 08:00:00 on 2021-06-01 with a partition unit of (YEAR), and the partition unit is changed to (MONTH) at 08:00:00 on 2021-06-02, then this change will only take effect after 00:00:00 on 2021-06-03. That is, data before 00:00:00 on 2021-06-03 will be stored in the year table data_table.2021, and data after 00:00:00 on 2021-06-03 will be stored in the month table data_table.202106.

[0089] like Figure 1 As shown, the index template `data_table` was created on October 5, 2018, with the partition unit being the year. On September 5, 2019, the partition unit was changed to the month. Therefore, data between October 5, 2018 and January 1, 2019 will be stored in the partitioned index table (year table) `data_table.2018`. Data between January 1, 2019 and September 6, 2019 will be stored in the partitioned index table (year table) `data_table.2019`. Data between September 6, 2019 and October 1, 2019 will be stored in the partitioned index table (month table) `data_table.201909`. Data between 2019.10.01 and 2019.11.01 will be stored in the partitioned index table (monthly table) data_table.201910, and so on.

[0090] After creating the index template using the above method, you can manage table partitioning based on the index template, such as... Figure 2 As shown, the specific table partitioning management process includes the following steps:

[0091] Step 101: Receive the user's data insertion request, which includes the business table name and the business data to be inserted.

[0092] Step 102: Determine the index template that matches the business table name. The index template includes the modification records of the table partitioning unit.

[0093] Upon receiving a data insertion request, the system searches for an index template that matches the business table name included in the data insertion request from all currently created index templates in the Elasticsearch platform. This example uses the first index template as an example to illustrate the determination of whether the first index template matches the business table name included in the data insertion request; the first index template is any index template currently created.

[0094] Specifically, the alias information and template matching pattern corresponding to the first index template are obtained from the first index template. It is then determined whether the alias information of the business table name matches the template matching pattern corresponding to the first index template. If yes, the first index template is determined to be the index template matching the business table name. If no, the first index template is determined not to match the business table name.

[0095] For each index template that has been created, the above method is used to determine the index template corresponding to the data insertion request.

[0096] Step 103: Based on the modification records of the business table name and the partition unit, insert the business data to be inserted into the target partition index table.

[0097] Retrieve all modification records for the partitioned table unit from the metadata of the index template matching the business table name. Obtain the timestamp corresponding to the data insertion request. If the data insertion request carries a timestamp corresponding to the business data to be inserted, retrieve the timestamp directly from the data insertion request. If the data insertion request does not carry a timestamp, determine the current time as the timestamp corresponding to the business data to be inserted.

[0098] Then, based on all modification records of the partition unit, the partition unit corresponding to the timestamp is determined. A partition table name is generated based on the business table name, timestamp, and the corresponding partition unit. The business data to be inserted is stored in the target partition index table corresponding to the partition table name. When storing the business data to be inserted in the target partition index table, a data identifier is also assigned to the business data to be inserted, and the mapping relationship between the data identifier and the business data to be inserted is stored. Users can then query the data based on the data identifier.

[0099] For example, if the timestamp corresponding to the data insertion request is 2018.08.10 13:01:05, and the modification records for the partition unit include [timestamp: 20180705, partition unit: year] and [timestamp: 20190905, partition unit: month], then the partition unit corresponding to this timestamp is year. Based on the business table name data-table, the timestamp 2018.08.10 13:01:05, and the partition unit "year" corresponding to this timestamp, a partition table named data-table.2018 is generated. The business data to be inserted is stored in the partition index table named data-table.2018.

[0100] After data is inserted based on the index template, it will be inserted into the corresponding partitioned index table. When a user needs to query a specific piece of data, they send a data query request to the Elasticsearch platform. This request includes the business table name, time parameters, and the data identifier of the data to be queried.

[0101] After receiving a user's data query request, the Elasticsearch platform retrieves the modification records of the partitioned table unit from the metadata of the index template that matches the business table name included in the query request. First, the index template matching the business table name is determined; the specific determination process is the same as in step 102 and will not be repeated here. After determining the matching index template, all modification records of the partitioned table unit are retrieved from the metadata of that index template.

[0102] Based on the business table name, time parameter, and modification records of the obtained table partitioning unit included in the data query request, the name of the partition table to be queried is generated. Then, based on the data identifier of the data to be queried, the data to be queried is retrieved from the partition index table corresponding to the partition table name.

[0103] In one implementation, the data query request includes a specific timestamp as the time parameter, such as 2018.08.10 13:01:05. Based on this timestamp and all modification records of the partition unit, the partition unit corresponding to this timestamp is determined. Then, based on the timestamp, its corresponding partition unit, and the business table name, the name of the partition table to be queried is generated, such as... Figure 3 As shown in the image. This implementation method enables accurate querying of data corresponding to a specified timestamp.

[0104] In another implementation, the data query request includes a time range as the time parameter, comprising the start and end timestamps of the query. First, the partition table names of all partitioned index tables are obtained from the index template matching the business table name. Then, the modification times of all modification records are obtained from the modification records of each partitioned unit. The start and end times, along with the modification times of all modification records, are sorted chronologically. Based on the modification times of the modification records located between the start and end timestamps after sorting, the partition table names for each time period between the start and end timestamps are determined from the partition table names of all partitioned index tables. A list of partition table names to be queried is generated based on the determined partition table names for each time period. Finally, based on the data identifier of the data to be queried, the data is retrieved from the partitioned index table corresponding to each partition table name in the list of partition table names.

[0105] To facilitate understanding, the following explanation is provided in conjunction with the accompanying drawings, such as... Figure 4As shown, the start timestamp is 2018-11-10 13:01:05, and the end timestamp is 2020-06-10 23:01:05. Based on the business table name `data_table`, retrieve the modification records of the table partitioning unit stored in the `_meta` field of the matching index template. All retrieved modification records are:

[0106]

[0107]

[0108] Based on the business table name data_table, retrieve the partition table names of all partition index tables in the matching index template: [data_table.2018,data_table.201911,data_table.2020,data_table.20200609,data_table.2021].

[0109] Retrieve the modification time of all modified records in the sub-table unit: ["20181005", "20190905", "20200215", "20200605", "20200612"]. Sort the start timestamp, end timestamp, and modification time of all modified records in chronological order to obtain the time series: ["20181005", "20181110", "20190905", "20200215", "20200605", "20200610", "20200612"].

[0110] Then, obtain the offset positions of the start and end timestamps within the time series, and calculate the actual partition index table for each sub-table within a unit period in sequence:

[0111] "20181110"--"20190905"(YEAR)-->Retrieve the year table: data_table.2018

[0112] "20190905"--"20200215"(MONTH)--> Retrieve the monthly table: data_table.201911

[0113] "20200215"--"20200605"(YEAR)-->Retrieve the year table: data_table.2020

[0114] "20200605"--"20200610"(DAY)--> Get the daily table: data_table.20200609

[0115] The final list of partition table names is [data_table.2018, data_table.201911, data_table.2020, data_table.20200609].

[0116] In this embodiment, the data in the partition index table can be flexibly deleted or modified based on the above-described data query method. To delete a piece of data, a data deletion request is sent to the ES platform. This request includes the business table name, a time parameter, and the data identifier of the data to be deleted. The ES platform receives the data deletion request and, based on the business table name and time parameter, generates a partition table name according to the method described above for determining the partition table name to be queried. Then, it deletes the data corresponding to the data identifier included in the data deletion request from the partition index table corresponding to the generated partition table name.

[0117] To modify a piece of data, a data modification request is sent to the Elasticsearch (ES) platform. This request includes the business table name, a time parameter, the data identifier of the data to be modified, and the new data. The ES platform receives this request and, based on the business table name and time parameter, generates a partition table name using the same method described above for determining the partition table name. Then, in the partition index table corresponding to the generated partition table name, the data corresponding to the data identifier included in the data modification request is replaced with the new data included in the request.

[0118] In the embodiments of this application, such as Figure 5 As shown, when a user first requests table creation, only an index template is created. The partitioned index table used to store the data is only created when data is inserted. Subsequent CRUD operations on the data are performed by first determining the corresponding partitioned table name through a partitioned table name preprocessing process, and then performing CRUD operations on the partitioned index table corresponding to that partitioned table name.

[0119] This application's embodiment creates an index template when a user first requests table creation. The metadata of this index template records modifications to the table partitioning unit, making modifications to the partitioning unit convenient and quick. Based on the table partitioning unit, flexible management of the partitioned index table is achieved, enabling CRUD operations on data in any partitioned index table. Through precise recording and management of modifications to the table partitioning unit, CRUD operations on business tables are quickly converted into operations on specific partitioned index tables, reducing the number of index shards processed per operation and thus improving data processing efficiency.

[0120] This embodiment provides a table partitioning management device based on an index template, which is used to execute the table partitioning management method based on an index template described in any of the above embodiments. Figure 6 As shown, the device includes:

[0121] The receiving module 201 is used to receive a user's data insertion request, which includes the business table name and the business data to be inserted.

[0122] The determination module 202 is used to determine the index template that matches the business table name. The index template includes the modification records of the table partitioning unit.

[0123] The data insertion module 203 is used to insert the business data to be inserted into the target partition index table based on the modification records of the business table name and the partition unit.

[0124] The device also includes: a template creation module for receiving a user's table creation request, which includes at least table partitioning unit and alias information; and creating an index template based on the table creation request, which includes table partitioning unit and alias information.

[0125] The template creation module is used to create index templates; generate template matching patterns corresponding to the index templates based on alias information; store the alias information and template matching patterns in the index templates; and store the modification records corresponding to the partition units in the metadata of the index templates based on the partition unit and the current time.

[0126] The device also includes: a table partitioning unit modification module, used to receive a table partitioning unit update request, which includes the business table name and the new table partitioning unit; generate a modification record corresponding to the new table partitioning unit based on the new table partitioning unit and the current timestamp; and store the modification record corresponding to the new table partitioning unit in the metadata of the index template corresponding to the business table name.

[0127] The determination module 202 is used to obtain the alias information and template matching pattern corresponding to the first index template from the first index template; the first index template is any index template that has been created; determine whether the alias information of the business table name and the first index template matches the template matching pattern corresponding to the first index template; if so, determine the first index template as the index template that matches the business table name.

[0128] The data insertion module 203 is used to obtain the modification record of the partition unit from the metadata of the index template that matches the business table name; obtain the timestamp corresponding to the data insertion request; determine the partition unit corresponding to the timestamp based on the modification record of the partition unit; generate the partition table name based on the business table name, timestamp and the partition unit corresponding to the timestamp; and store the business data to be inserted into the target partition index table corresponding to the partition table name.

[0129] The device also includes: a data query module, used to receive user data query requests, the data query requests including business table name, time parameters, and data identifier of the data to be queried; based on the business table name, to obtain the modification records of the partition unit from the metadata of the index template matching the business table name; based on the business table name, time parameters, and modification records of the partition unit, to generate the name of the partition table to be queried; and based on the data identifier of the data to be queried, to query the data to be queried from the partition index table corresponding to the name of the partition table to be queried.

[0130] The time parameters include the start and end timestamps to be queried; the data query module is used to obtain the partition table names of all partitioned index tables from the index template matching the business table name; obtain the modification time of all modification records from the modification records of the partition unit; sort the start timestamp, end timestamp, and modification times of all modification records in chronological order; based on the modification times of the modification records located between the start and end timestamps after sorting, determine the partition table names of the partitioned index tables for each time period between the start and end timestamps from the partition table names of all partitioned index tables; and generate a list of partition table names to be queried based on the determined partition table names corresponding to each time period.

[0131] This application's embodiment creates an index template when a user first requests table creation. The metadata of this index template records modifications to the table partitioning unit, making modifications to the partitioning unit convenient and quick. Based on the table partitioning unit, flexible management of the partitioned index table is achieved, enabling CRUD operations on data in any partitioned index table. Through precise recording and management of modifications to the table partitioning unit, CRUD operations on business tables are quickly converted into operations on specific partitioned index tables, reducing the number of index shards processed per operation and thus improving data processing efficiency.

[0132] Please refer to the following. Figure 7 This illustrates a schematic diagram of an electronic device provided by some embodiments of this application. For example... Figure 7 As shown, the electronic device 7 includes: a processor 700, a memory 701, a bus 707, and a communication interface 703. The processor 700, the communication interface 703, and the memory 701 are connected via the bus 707. The memory 701 stores a computer program that can run on the processor 700. When the processor 700 runs the computer program, it executes the table partitioning management method based on the index template provided in any of the foregoing embodiments of this application. The electronic device may be an electronic device with a touch-sensitive display.

[0133] The memory 701 may include high-speed random access memory (RAM) or non-volatile memory, such as at least one disk storage device. Communication between this system network element and at least one other network element is achieved through at least one communication interface 703 (which can be wired or wireless), such as the Internet, wide area network, local area network, or metropolitan area network.

[0134] Bus 707 can be an ISA bus, PCI bus, or EISA bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. Memory 701 is used to store programs. After receiving an execution instruction, processor 700 executes the program. The index template-based table management method disclosed in any of the foregoing embodiments of this application can be applied to processor 700, or implemented by processor 700.

[0135] The processor 700 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed by the integrated logic circuitry in the hardware of the processor 700 or by instructions in software form. The processor 700 may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), an off-the-shelf programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules may reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in memory 701. Processor 700 reads the information in memory 701 and, in conjunction with its hardware, completes the steps of the above method.

[0136] The electronic device provided in this application embodiment and the table partitioning method based on index templates provided in this application embodiment are based on the same application concept and have the same beneficial effects as the methods they adopt, run or implement.

[0137] This application also provides a computer-readable storage medium corresponding to the index template-based table management method provided in the foregoing embodiments. Please refer to [link / reference]. Figure 8 , Figure 8 The computer-readable storage medium shown is an optical disc 30, on which a computer program (i.e., a program product) is stored. When the computer program is run by a processor, it executes the table partitioning method based on the index template provided in any of the foregoing embodiments.

[0138] In addition, examples of the computer-readable storage medium may include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other optical and magnetic storage media, which will not be described in detail here.

[0139] The computer-readable storage medium provided in the above embodiments of this application and the table partitioning method based on index templates provided in the embodiments of this application are based on the same application concept and have the same beneficial effects as the methods adopted, run or implemented by the applications stored therein.

[0140] It should be noted that the algorithms and displays provided herein are not inherently related to any particular computer, virtual device, or other equipment. Various general-purpose devices can also be used in conjunction with the teachings herein. Based on the above description, the required structure for constructing such devices is obvious. Furthermore, this application is not directed to any particular programming language. It should be understood that the content of this application described herein can be implemented using various programming languages, and the above description of specific languages ​​is for the purpose of disclosing the best mode of implementation of this application.

[0141] Similarly, it should be understood that, for the purpose of simplification and aiding understanding of one or more aspects of the application, various features of the application are sometimes grouped together in a single embodiment, figure, or description thereof in the above description of exemplary embodiments of the application. However, this disclosure should not be construed as reflecting an intention that the claimed application requires more features than expressly recited in each claim. Rather, as reflected in the following claims, the application aspect comprises fewer than all features of a single foregoing disclosed embodiment. Therefore, the claims following the detailed description are hereby expressly incorporated into that detailed description, wherein each claim itself is a separate embodiment of the application.

[0142] The various component embodiments of this application can be implemented in hardware, or as software modules running on one or more processors, or a combination thereof. Those skilled in the art will understand that microprocessors or digital signal processors (DSPs) can be used in practice to implement some or all of the functions of some or all of the components in the virtual machine creation apparatus according to embodiments of this application. This application can also be implemented as a device or apparatus program for performing part or all of the methods described herein. The program implementing this application can be stored on a computer-readable medium, or can be in the form of one or more signals. Such signals can be downloaded from an Internet website, provided on a carrier signal, or provided in any other form.

[0143] The above description is merely a preferred embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An index template-based subtable management method, characterized by, The method includes: Receive a user's data insertion request, the data insertion request including the business table name and the business data to be inserted; Determine an index template that matches the business table name. The metadata of the index template includes modification records of the table partitioning unit, which is the partitioning unit used to divide the partitioned index table based on the index template. Based on the modification records of the business table name and the partition unit, the business data to be inserted is inserted into the target partition index table; wherein, a partition table name is generated based on the business table name, the timestamp corresponding to the data insertion request, and the partition unit corresponding to the timestamp; the business data to be inserted is stored in the target partition index table corresponding to the partition table name, and the target partition index table is created when inserting data.

2. The method of claim 1, wherein, Before determining the index template that matches the business table name, the method further includes: Receive a user's table creation request, which includes at least table partitioning unit and alias information; Based on the table creation request, an index template is created, which includes the table partitioning unit and the alias information.

3. The method of claim 2, wherein, The step of creating an index template based on the table creation request includes: Create an index template; Based on the alias information, generate the template matching pattern corresponding to the index template; The alias information and the template matching pattern are stored in the index template; Based on the partition unit and the current time, the modification record corresponding to the partition unit is stored in the metadata of the index template.

4. The method according to any one of claims 1 to 3, characterized in that, The method further includes: Receive a table partitioning unit update request, the table partitioning unit update request including the business table name and the new table partitioning unit; Based on the new table partitioning unit and the current timestamp, generate the modification record corresponding to the new table partitioning unit; The modification records corresponding to the new table partitioning unit are stored in the metadata of the index template corresponding to the business table name.

5. The method according to claim 1, characterized in that, The step of determining the index template that matches the business table name includes: Obtain the alias information and template matching pattern corresponding to the first index template; the first index template is any index template that has been created so far. Determine whether the alias information of the business table name and the first index template matches the template matching pattern corresponding to the first index template; If so, the first index template is determined to be the index template that matches the business table name.

6. The method according to claim 1, characterized in that, The step of inserting the business data to be inserted into the target partition index table based on the modification records of the business table name and the partition unit includes: Retrieve the modification records of the table partitioning unit from the metadata of the index template that matches the business table name; Obtain the timestamp corresponding to the data insertion request; Based on the modification records of the table sub-unit, determine the table sub-unit corresponding to the timestamp; Generate a partition table name based on the business table name, the timestamp, and the partition unit corresponding to the timestamp; The business data to be inserted is stored in the target partition index table corresponding to the partition table name.

7. The method according to any one of claims 1-6, characterized in that, The method further includes: Receive a user's data query request, the data query request including the business table name, time parameter and data identifier of the data to be queried; Based on the business table name, retrieve the modification records of the table partitioning unit from the metadata of the index template that matches the business table name; Based on the modification records of the business table name, the time parameter, and the table sub-unit, generate the partition table name to be queried; Based on the data identifier of the data to be queried, the data to be queried is retrieved from the partition index table corresponding to the partition table name to be queried.

8. The method according to claim 7, characterized in that, The time parameters include the start and end timestamps to be queried; the step of generating the partition table name to be queried based on the business table name, the time parameters, and the modification records of the partition unit includes: Retrieve the partition table names of all partition index tables from the index templates that match the business table names; Obtain the modification time of all modification records from the modification records of the aforementioned table unit; Sort the start timestamp, the end timestamp, and the modification times of all modified records in chronological order. Based on the modification time of the modification records located between the start timestamp and the end timestamp after sorting, determine the partition table name of the partition index table for each time period between the start timestamp and the end timestamp from the partition table names of all partition index tables; Based on the partition table names corresponding to each determined time period, generate a list of partition table names to be queried.

9. A table partitioning management device based on an index template, characterized in that, The device includes: The receiving module is used to receive a user's data insertion request, which includes a business table name and the business data to be inserted. The determination module is used to determine the index template that matches the business table name. The metadata of the index template includes modification records of the table partitioning unit, which is the partitioning unit used to divide the partitioned index table based on the index template. The data insertion module is used to insert the business data to be inserted into the target partition index table according to the business table name and the modification record of the partition unit; The device is further configured to generate a partition table name based on the business table name, the timestamp corresponding to the data insertion request, and the partition unit corresponding to the timestamp; and to store the business data to be inserted into the target partition index table corresponding to the partition table name, wherein the target partition index table is created when inserting data.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, The processor executes the computer program to implement the method as described in any one of claims 1-8.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1-8.

Images