Database management method and device and storage medium

Abstract

The invention discloses a database management method and device and a storage medium, and the method comprises the steps: determining the cycle length of a first index based on a data life cycle feature corresponding to the first index; the number of fragments of the first index in a target period is determined, and the target period is a current period or a future period; and creating the first index corresponding to the target period in the database according to the fragment number and the period length. In this way, the resource storage efficiency of the database can be improved.

Description

Technical Field

[0001] This application relates to the field of database technology, specifically to a database management method, device, and storage medium. Background Technology

[0002] Data searching in existing databases is typically performed using indexes. However, the management methods for existing database indexes have certain limitations. Taking Elasticsearch (ES, a distributed search and data analytics engine) as an example, ES is built on Apache Lucene (Lucene, an open-source, high-performance full-text search library) and is designed specifically for handling large-scale data. It provides near real-time full-text search capabilities, supports indexing and retrieval of both structured and unstructured data, and is widely used in log analysis, search engines, data monitoring, and recommendation systems. In these scenarios, the data types are diverse, including time-sensitive logs and monitoring data, as well as core business data that needs to be retained for a long time, generally exhibiting distinct lifecycle characteristics.

[0003] However, with the explosive growth of various types of data, traditional ES index management methods have problems such as the lack of an automatic mechanism for cleaning up expired data, which leads to the long-term accumulation of invalid data, causing index bloat, which not only wastes a lot of storage resources, but also significantly reduces query response efficiency. Summary of the Invention

[0004] To address the aforementioned technical problems, this application provides a database management method, device, and storage medium to improve storage resource utilization.

[0005] According to one embodiment of this application, a database management method is provided, comprising: The period length of the first index is determined based on the data lifecycle characteristics corresponding to the first index. Determine the number of shards for the first index in the target period, wherein the target period is the current period or a future period; Based on the number of shards and the period length, create the first index corresponding to the target period in the database.

[0006] To solve the above-mentioned technical problems, one technical solution adopted in this application is to provide an electronic device, including a memory and a processor, wherein the memory is used to store a computer program, and when the computer program is executed by the processor, it is used to implement the database management method in the above-mentioned technical solution.

[0007] To solve the above-mentioned technical problems, one technical solution adopted in this application is to provide a computer-readable storage medium for storing a computer program, which, when executed by a processor, is used to implement the database management method in the above-mentioned technical solution.

[0008] Through the above scheme, the database management method provided in this application determines the period length of the first index and the number of shards of the first index in the target period based on the data lifecycle characteristics corresponding to the first index. Finally, the first index corresponding to the target period is created in the database according to the number of shards and the period length. In this way, the index period length fits the data lifecycle characteristics, avoids invalid data occupying storage resources for a long time, and thus improves the utilization rate of database storage resources. Attached Figure Description

[0009] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. Wherein: Figure 1 This is a schematic diagram of an ES architecture provided in this application; Figure 2 This is a flowchart illustrating an embodiment of the database management method provided in this application; Figure 3 This is a schematic diagram of an attribute diagram provided in this application; Figure 4 This is a flowchart illustrating another embodiment of the database management method provided in this application; Figure 5 This is a schematic diagram of the structure of an embodiment of the electronic device provided in this application; Figure 6 This is a schematic diagram of an embodiment of the computer-readable storage medium provided in this application. Detailed Implementation

[0010] The present application will now be described in further detail with reference to the accompanying drawings and embodiments. It should be particularly noted that the following embodiments are for illustrative purposes only and do not limit the scope of the application. Similarly, the following embodiments are only some, not all, embodiments of the present application, and all other embodiments obtained by those skilled in the art without inventive effort are within the scope of protection of the present application.

[0011] In this application, the reference to "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0012] It should be noted that the terms "first," "second," etc., used in this application are for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined with "first," "second," etc., may explicitly or implicitly include at least one of that feature. In the description of this application, "multiple" means at least two, such as two, three, etc., unless otherwise explicitly specified. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or apparatuses.

[0013] In the database field, an index is a data structure used to accelerate data retrieval. Its core function is to pre-organize key information of the data, such as field values ​​and positions, so that the database does not need to scan the entire data during the query, thereby greatly improving retrieval efficiency.

[0014] Taking Elasticsearch (ES) as an example, the ES index is the core logical unit for storing, managing, and retrieving data in Elasticsearch. Essentially, it is a collection of JSON documents with similar structures (similar to "tables" in relational databases). It has a built-in inverted index structure to accelerate queries, and at the same time, it achieves distributed data storage and high availability through sharding and replication mechanisms. It is the basic carrier for ES to achieve efficient data processing.

[0015] Please see Figure 1 , Figure 1This is a schematic diagram of an Elasticsearch (ES) architecture provided in this application. The core elements include a cluster, nodes, primary shards (P), replica shards (R), and clients. A cluster is a distributed system composed of multiple nodes. All nodes within the cluster work collaboratively, sharing data and providing a unified service entry point. A node is a single server instance running ES. The diagram shows three nodes (Node1, Node2, and Node3) as the basic building blocks of the cluster. Node1, labeled *Master, represents the master node of the cluster, responsible for core control logic such as metadata management, node status monitoring, and shard allocation. P0, P1, and P2 are primary shards, the original storage units for data. ES splits index data into multiple primary shards and distributes them across different nodes, achieving horizontal scaling and parallel processing. R0, R1, and R2 are replica shards, redundant backups of the primary shards, improving data availability and distributing query pressure. Multiple clients interact with nodes in the cluster via the network, sending requests for data writing, querying, and management. The inverted index technology based on Lucene is the core support for the efficient retrieval of Elasticsearch. Its core advantage lies in splitting text data into terms and establishing a mapping relationship between "terms and document IDs". Therefore, it can quickly locate matching content without traversing the entire data. Thus, it can efficiently support complex retrieval needs such as Boolean queries, fuzzy queries, and phrase queries. It is suitable for scenarios that require fast and accurate retrieval of large amounts of text data, such as search engines, knowledge bases, and document management systems.

[0016] Elasticsearch (ES) has a wide range of applications, covering diverse fields such as log analysis, full-text search, time-series data monitoring, enterprise knowledge base retrieval, and e-commerce product recommendation. It can also complement various databases. For example, when working with relational databases such as MySQL (an open-source, high-performance relational database management system), it can compensate for the shortcomings of MySQL in full-text search and rapid querying of massive amounts of data. When working with time-series databases, it can enhance the aggregation analysis and visualization capabilities of time-series data. When working with graph databases, it can focus on attribute retrieval and efficient filtering of complex related data.

[0017] However, when Elasticsearch indexes are used in conjunction with relational, time-series, and graph databases, problems such as index bloat, poor sharding adaptation, and insufficient data consistency often occur due to a lack of targeted management. This application provides a database management method that, based on the data lifecycle characteristics corresponding to a first index, determines the period length of the first index and the number of shards for the first index in the target period. Finally, it creates the first index corresponding to the target period in the database according to the number of shards and the period length. This ensures that the index period length aligns with the data lifecycle characteristics, avoids invalid data occupying storage resources for extended periods, and thus improves the utilization rate of database storage resources.

[0018] Please see Figure 2 , Figure 2 This is a flowchart illustrating an embodiment of the database management method provided in this application. It should be noted that if substantially the same result is achieved, this embodiment does not necessarily replace it. Figure 2 The illustrated process sequence is limited. For example... Figure 2 As shown, this embodiment includes: S110: Determine the period length of the first index based on the data lifecycle characteristics corresponding to the first index.

[0019] Data lifecycle (TTL) refers to the complete process from acquisition or creation, storage, query or analysis, archiving to final deletion or cleanup. It is the stage trajectory of data in business flow. Different types of data have significantly different lifecycle stages and durations due to different business needs.

[0020] Data lifecycle characteristics refer to the regularities and inherent attributes exhibited by data at each stage of its lifecycle. They reflect the dynamic changes in data and the decay patterns of business value. Data lifecycle characteristics include access frequency, lifespan, concentration of data expiration time, and data volume change trends. Among them, access frequency refers to the frequency of data being queried or accessed during its lifecycle; lifespan is the duration for which data needs to be effectively stored and guaranteed to be available; concentration of data expiration time describes whether a large amount of data reaches the end of its lifecycle at a similar time point; and data volume change trends refer to the pattern of growth or decrease in the total amount of data during its lifecycle.

[0021] Based on the data lifecycle characteristics corresponding to the first index, the cycle length of the first index is determined so that the index and the data value cycle are accurately matched, avoiding index bloat caused by long-term accumulation of invalid data, and reducing invalid traversal during queries to ensure read and write performance.

[0022] In one embodiment, based on the data lifecycle characteristics corresponding to the first index, it is determined whether the target data corresponding to the first index belongs to a first data type or a second data type. The first data type includes at least one of a high-frequency access data type, a low-lifetime data type, and a high-frequency access data type with a low lifetime. The second data type includes at least one of a low-frequency access data type, a high-lifetime data type, and a low-frequency access data type with a high lifetime.

[0023] In response to the target data belonging to a first data type, the period length of the first index is selected from the first period length set; in response to the target data belonging to a second data type, the period length of the first index is selected from the second period length set. Each period length in the first period length set is shorter than each period length in the second period length set.

[0024] In one implementation, complete lifecycle characteristic data of the first index target data is collected to clarify the access frequency, business-required lifespan, concentrated expiration, and data volume increase / decrease patterns at different stages. Combining business needs such as query efficiency and compliance retention requirements, as well as resource constraints, the index period granularity is dynamically selected through adaptive analysis of the characteristic data. High-frequency, short-term, and concentratedly expired data can be set to shorter periods, while low-frequency, long-term data can be set to longer periods. Finally, the index period granularity adapted to the data characteristics is determined, such as daily, weekly, monthly, or grade-level, ensuring that the period length covers the high-value stages of the data and avoids ineffective and redundant stages. For example, daily or weekly indexes can be used for relationships of intimacy between people or confidence relationships between people and objects, while monthly or grade-level indexes can be used for basic information such as people and vehicles. By comprehensively considering factors such as data update frequency, query patterns, and storage costs, it can be ensured that the index partitioning meets both business timeliness requirements and avoids index fragmentation caused by excessively fine periods.

[0025] In one embodiment, indexes for the target period can be created in advance through scheduled tasks, and storage quotas and node allocation strategies can be planned for the indexes in advance. Furthermore, when the indexes are pre-created, strategies such as automatic expiration, archiving, and deletion can be configured in conjunction with data lifecycle management to optimize storage utilization.

[0026] In one example, if the target period is a future period, and the period length of the first index belongs to the first period length set, the first index corresponding to the future period is created a first number of periods in advance; if the period length of the first index belongs to the second period length set, the first index corresponding to the future period is created a second number of periods in advance. Here, each period length in the first period length set is shorter than each period length in the second period length set, and the first number is less than the second number.

[0027] In one specific implementation, for short-term cycles such as daily or weekly, pre-creation can be performed one cycle in advance. For example, the log index for the current day can be created one day in advance. This ensures that the index has completed preparation work such as shard initialization and mapping definition when data is written, avoiding performance fluctuations during automatic index creation and ensuring low latency and high throughput for data writing. For long-term cycles such as monthly or grade-based, pre-creation can be performed several cycles in advance. For example, monthly indexes can be created three months in advance. This allows sufficient time for fine-grained planning of the number of shards and replica distribution, adapting to the long-term growth trend of data volume and avoiding the overhead of shard rebalancing caused by later expansion.

[0028] S120: Determine the number of shards for the first index in the target period.

[0029] The target period includes the current period or a future period, and the number of shards refers to the number of shards that the first index is divided into within the target period. Determining the number of shards for the first index in the target period ensures that the pre-created index is adapted to the data characteristics and business needs of the target period from the initialization stage, avoiding performance, resource, or stability issues caused by unreasonable sharding configuration later.

[0030] In one embodiment, if the target period is a future period, the number of shards for the first index in the future period is determined based on the historical period index data volume of the first index. In another embodiment, if the database currently has a first index corresponding to multiple periods, the historical period index data volume is the largest index data volume among the first index data volumes corresponding to each of the multiple periods; if the database currently has only a first index corresponding to a single period, the historical period index data volume is the first index data volume corresponding to that single period. For example, if the first index has at least two historical periods, the maximum value of the index data volume in the at least two historical periods is used as the first index data volume; if the first index has only had one historical period in the past, the index data volume corresponding to that historical period is directly used as the first index data volume.

[0031] In one embodiment, the number of shards for the first index in future periods can be determined by the database cluster hardware configuration and the preferred shard size. In one example, a reference number of shards is determined based on the database cluster hardware configuration. Using the reference number of shards, several candidate number of shards are generated. Based on the historical period index data volume of the first index, the number of shards for the first index in future periods is selected from the several candidate number of shards.

[0032] In one implementation, the cluster hardware configuration includes cluster physical nodes and node storage configuration. The product of the number of cluster physical nodes and the node storage configuration can be used to determine the reference number of shards. For example, considering the physical node size of the ES cluster, such as 3 independent servers, and the node storage configuration, such as 2 solid-state drives (SSDs) deployed per node, the minimum number of primary shards for the first index is set to 3. This achieves a basic allocation pattern of 1 primary shard corresponding to 1 node, avoiding load imbalance where some nodes carry multiple primary shards while others have no primary shards. The product of the cluster physical nodes and the node storage configuration, i.e., 3 nodes × 2 disks = 6 disks, is then determined as the reference number of shards.

[0033] In one embodiment, the product of the reference shard number and multiple coefficients can be used as multiple candidate shard numbers, corresponding to different types of shard number selection strategies, so as to dynamically select the shard number that is compatible with the future period of the first index. For example, the shard number selection strategy may include a conservative strategy (P1), a balanced strategy (P2), and an aggressive strategy (P3). In this embodiment, the corresponding coefficients are set to k1=1 / 2, k2=1, and k3=3 / 2, respectively. Multiplying the reference shard number by multiple coefficients, multiple candidate shard numbers are obtained as P1=3, P2=6, and P3=9, respectively.

[0034] In one embodiment, the ratio of the historical period index data volume to the number of candidate shards is used as the single shard size corresponding to each candidate shard number. Following the ascending order of the number of candidate shards, the first candidate shard with a single shard size within a preset size range is selected as the shard number for the first index in the future period. Alternatively, the candidate shard with a single shard size closest to the ideal size is selected as the shard number for the first index in the future period. For example, assuming the single shard size is V, the historical period index data volume is S, the number of candidate shards is P, and the preset size range is [5GB, 20GB], the single shard volume corresponding to each group of candidate shards (P1=3, P2=6, P3=9) is calculated according to V=S / P. The candidate shards are checked sequentially in ascending order, i.e., P1→P2→P3, and the first candidate shard with a single shard size within the preset size range [5GB, 20GB] is selected as the shard number for the first index in the future period. In some examples, there may be cases where no candidate shard number corresponds to a single shard size that meets the preset size range. In such cases, the candidate shard number whose single shard size is closest to the ideal size (e.g., 10GB) can be selected as the first index for the number of shards in future periods, so that the shard size approaches the ideal value.

[0035] S130: Create the first index corresponding to the target period in the database according to the number of shards and the period length.

[0036] Based on the pre-determined target period (current or future period) and its corresponding period length (e.g., daily or monthly), clarify the index naming conventions. Then, combined with the planned number of shards, including the number of cluster nodes to adapt to, storage configuration, and data volume estimates, synchronously configure the number of primary shards, replicas, storage node allocation, lifecycle policies, and field mapping rules. Subsequently, the creation operation can be performed through the database API or visualization tools. Finally, verify the index status and shard distribution. For the current period index, additional read / write performance testing and synchronization of existing data are required. For future period indexes, confirmation of initialization is sufficient. This ensures that the first index can adapt to the data characteristics and cluster resources of the target period from the very beginning of its creation.

[0037] In one implementation, if the target period is the current period, data writing should be initiated immediately after the index is confirmed to be enabled, such as in a real-time log collection scenario, where the data writing channel is directly bound after creation. In another implementation, if the target period is a future period, the pre-creation timing is determined based on the period type of the first index. After creation, the index can be set to an "inactive" state, which will be automatically activated when the period begins, thus avoiding premature occupation of read and write resources.

[0038] In one embodiment, the first index has a time-to-live (TTL) setting, and the database may also include a second index without a TTL setting. For the second index without a TTL, the number of shards for the second index is redefined. A new second index is created based on the redefined number of shards. After migrating the data from the original second index to the new second index, the original second index is deleted. For example, when an index in Elasticsearch without a TTL causes performance degradation due to excessively large shards, the number of shards is dynamically adjusted using the reindex method to achieve smooth data migration and avoid query latency, write blocking, or cluster load imbalance caused by unreasonable sharding.

[0039] In one embodiment, based on the lifespan and creation time of the first index corresponding to the target period, it is determined whether the first index corresponding to the target period has expired. In response to the expiration of the first index corresponding to the target period and the fact that the first index corresponding to the target period is not currently being actively queried, the first index corresponding to the target period is asynchronously cleaned up. The thread used to clean up the first index has a lower priority than the thread used for data writing to the database. For example, assuming that in a certain Elasticsearch cluster, the TTL of a daily index is configured to automatically expire 7 days after creation, when an index reaches the 7-day expiration threshold, the cluster automatically triggers a deletion task. This task is submitted to a preset low-priority thread pool (with a lower priority than the real-time data writing thread pool). Subsequently, the system verifies the index shard status through the API (Application Programming Interface), calls the active query detection interface to confirm that no business queries reference the index, and verifies that no data writing tasks are associated. Then, the low-priority thread pool performs the index cleanup operation during idle periods of cluster I / O (Input / Output) resources, releasing SSD (Solid State Drive) storage space. This ensures that the entire process does not compete with real-time log writing or business queries for I / O, thus guaranteeing the stable operation of core business processes.

[0040] The cleanup of the first index corresponding to the target period can be optimized through several strategies. In one implementation, cleanup of the first index corresponding to the target period includes merging multiple expired index segments before cleanup. For example, for multiple expired first indexes (such as a batch of daily expired log indexes), a segment merging operation is first triggered, that is, merging the scattered small data segments within each index into a few large segments to reduce disk fragmentation and metadata usage. Then, the cleanup is performed in batches, which improves the efficiency of storage space release and avoids storage waste caused by residual small files.

[0041] In one implementation, if the data volume of the first index corresponding to the target period exceeds a preset data volume, asynchronous cleanup of the first index corresponding to the target period is performed when the cluster load of the database is at a low point. For example, if the data volume of the first index in the target period exceeds a preset threshold (e.g., a single index exceeds 500GB), it is first marked as pending cleanup. The monitoring tool captures the low-load period of the cluster (e.g., 2-4 AM, when the real-time write and query request volume drops by more than 60%), and then the asynchronous cleanup task is started. That is, the task is executed in an independent thread in the background, without blocking the real-time data processing of the main thread. At the same time, the I / O and CPU (central processing unit) usage of the cleanup process is limited through a resource isolation mechanism to ensure that the cleanup of large-volume indexes does not cause cluster load fluctuations, thus balancing storage release and business stability.

[0042] To better illustrate the database management method of this application, this embodiment provides a graph database management method that coordinates with Elasticsearch (ES). However, the use of ES is not limited to this scenario and can be flexibly adapted according to different business data characteristics, so no limitation is made here.

[0043] Graph databases are a new type of NoSQL (a general term for a class of non-relational data storage systems) database based on graph theory. They directly store the relationships between data in the form of nodes (Vetex), edges (Edge), and properties (Properties). Their data storage and query logic are designed around graph theory, exhibiting high performance in complex relationship processing scenarios. Therefore, they are widely used in social networks, recommendation systems, artificial intelligence, knowledge graphs, financial transactions, and other fields. Please refer to [link / reference]. Figure 3 , Figure 3 This is a schematic diagram of an attribute graph provided in this application. An attribute graph is a graph structure composed of vertices, edges, properties, and labels, naturally representing entities, relationships, and related features in the real world, such as... Figure 3 The attribute graph in the image represents the relationship and attributes between two characters (Kyle and EHang) and a test item.

[0044] JanusGraph (an open-source distributed graph database engine) is developed based on the Tinkerpop framework (an open-source graph computing framework and standardized ecosystem), uses the Gremlin query language (a general-purpose graph traversal and query language provided by graph computing frameworks), and follows an attribute graph model design. It requires external storage systems such as Elasticsearch, Apache Cassandra (an open-source distributed column-family NoSQL database), or HBase (an open-source distributed column-family NoSQL database) to store vertex and edge information. Elasticsearch is often used for graph data indexing to support efficient full-text search, range queries, and complex condition filtering. JanusGraph also possesses core functionalities such as graph model management, transaction coordination, cache optimization, and decision-making regarding storage and index access timing, supporting the storage and query needs of large-scale graph data.

[0045] Please see Figure 4 , Figure 4 This is a flowchart illustrating another embodiment of the database management method provided in this application. It should be noted that if substantially the same result is achieved, this embodiment does not necessarily reflect that outcome. Figure 4 The illustrated process sequence is limited. For example... Figure 4 As shown, this embodiment includes: The Load-Schema process begins by loading the graph's data schema using a Gremlin script, defining the types, attributes, and indexing rules for vertices and edges. During the schema import phase, the Time-To-Live (TTL) configurations of the graph's vertices, edges, and their attributes are parsed. By statistically analyzing their timeliness distribution characteristics, such as the concentration of data expiration times and data volume trends, the periodicity of the index is dynamically selected (e.g., high-frequency short-term data, low-frequency long-term data), determining the current index type as daily, weekly, monthly, or grade-level, thus obtaining the periodic index type.

[0046] The system initiates a dual-parallel process. On one hand, it first creates the initial index for the current period to handle real-time data writes, while simultaneously acquiring the total disk capacity of the Elasticsearch cluster to assess storage resources. If the index for future periods has not yet been created, it checks whether historical period indexes (such as the last two periods) exist. If two historical period indexes exist, the maximum data volume is used as a benchmark to predict the data volume for future periods. If only one historical period exists, the data volume for future periods is predicted based on the index size of the previous period. Based on this, the number of shards and storage quotas are planned to ensure that the pre-created future indexes adapt to data growth.

[0047] Furthermore, based on the predicted data volume and coefficients for future periods, the single-shard volume corresponding to each coefficient (coefficient 1 / 2, coefficient 1, coefficient 3 / 2) is calculated and judged sequentially according to V=S / P to determine whether it is within [5GB, 10GB]. If the shard volume corresponding to a certain coefficient is within the preset range [5GB, 10GB], the candidate shard number corresponding to the current coefficient is determined as the number of shards for creating the index in the next period; otherwise, the judgment continues sequentially. If the single-shard volume corresponding to each coefficient is not within [5GB, 10GB], the candidate shard number closest to 10GB is taken as the primary shard number for the index in the next period. Finally, the determined number of shards is used to create the index for the next period, and the field mapping of the index in the next future period remains consistent with the historical index.

[0048] On the other hand, expired index cleanup can be triggered by scheduled tasks (such as every day at midnight). First, the current period and the most recently expired period (such as historical indexes older than 30 days in a daily index) are calculated, and then the expired index cleanup operation is performed to free up disk space. A deletion time can also be preset to ensure that the expired index cleanup operation is triggered and executed within a reasonable timeframe. If the preset deletion time has been exceeded, the deletion task is stopped and will continue to be executed next time.

[0049] In this embodiment, the entire database management process, through automated index creation and expiration cleanup, can ensure efficient querying of graph data, avoid redundant storage resources, and adapt to the long-term management needs of large-scale graph data.

[0050] Please see Figure 5 , Figure 5 This is a schematic diagram of the structure of an embodiment of the electronic device provided in this application. The electronic device 60 includes a memory 61 and a processor 62 that are interconnected. The memory 61 is used to store computer programs. When the computer programs are executed by the processor 62, they are used to implement the database management method in the above embodiment.

[0051] The methods described in the above embodiments can exist in the form of a computer program; therefore, this application proposes a computer-readable storage medium. Please refer to [link / reference needed]. Figure 6 , Figure 6 This is a schematic diagram of an embodiment of a computer-readable storage medium provided in this application. The computer-readable storage medium 80 is used to store a computer program 81, which can be executed to implement the database management method in the above embodiment.

[0052] The computer-readable storage medium 80 can be any medium capable of storing program code, such as a server, USB flash drive, external hard drive, read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0053] The above description is merely an embodiment of this application and does not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A database management method, characterized in that, The method includes: The period length of the first index is determined based on the data lifecycle characteristics corresponding to the first index. Determine the number of shards for the first index in the target period, wherein the target period is the current period or a future period; Based on the number of shards and the period length, create the first index corresponding to the target period in the database.

2. The method according to claim 1, characterized in that, The target period is a future period; determining the number of shards for the first index in the target period includes: Based on the historical periodic index data volume of the first index, determine the number of shards for the first index in future periods.

3. The method according to claim 2, characterized in that, When the database currently has multiple first indexes corresponding to multiple periods, the historical period index data volume is the maximum index data volume among the first index data volumes corresponding to each of the multiple periods; If the database currently contains only a first index corresponding to a single period, the amount of historical period index data is the amount of first index data corresponding to a single period.

4. The method according to claim 2 or 3, characterized in that, Determining the number of shards for the first index in future periods based on the historical periodic index data volume of the first index includes: Based on the cluster hardware configuration of the database, determine the reference number of shards; Using the reference number of fragments, generate several candidate fragment numbers; Based on the historical periodic index data volume of the first index, the number of shards for the first index in the future period is selected from the plurality of candidate shard numbers.

5. The method according to claim 4, characterized in that, The cluster hardware configuration includes cluster physical nodes and node storage configuration; determining the reference number of shards based on the cluster hardware configuration of the database includes: The product of the number of physical nodes in the cluster and the node storage configuration is determined as the reference number of shards; And / or, the step of generating a plurality of candidate shard numbers using the reference shard number includes: The product of the reference number of fragments with multiple coefficients is used as multiple candidate number of fragments; And / or, selecting the number of shards for the first index in future periods from the plurality of candidate shard numbers based on the historical periodic index data volume of the first index includes: The ratio of the historical periodic index data volume to the number of each candidate shard is used as the single shard size corresponding to each candidate shard number. According to the order of the number of candidate fragments from smallest to largest, the first candidate fragment with a single fragment size within a preset size range is selected as the number of fragments for the first index in the future period; or, the candidate fragment with a single fragment size closest to the ideal size is selected as the number of fragments for the first index in the future period.

6. The method according to claim 1, characterized in that, The data lifecycle characteristics include the access frequency, survival time, data expiration time concentration, and data volume change trend of the target data corresponding to the first index; And / or, determining the period length of the first index based on the data lifecycle characteristics corresponding to the first index includes: Based on the data lifecycle characteristics corresponding to the first index, it is determined that the target data corresponding to the first index belongs to a first data type or a second data type. The first data type includes at least one of high access frequency data type, low survival time data type, and high access frequency and low survival time data type. The second data type includes at least one of low access frequency data type, high survival time data type, and low access frequency and high survival time data type. In response to the target data belonging to the first data type, the period length of the first index is selected from the first period length set; In response to the target data belonging to the second data type, the period length of the first index is selected from the second period length set, wherein each period length in the first period length set is shorter than each period length in the second period length set.

7. The method according to claim 1, characterized in that, The target period is a future period; when the period length of the first index belongs to the first period length set, the first index corresponding to the future period is created in advance by a first number of periods; when the period length of the first index belongs to the second period length set, the first index corresponding to the future period is created in advance by a second number of periods, wherein each period length in the first period length set is shorter than each period length in the second period length set, and the first number is less than the second number.

8. The method according to claim 1, characterized in that, The first index has a time-to-live setting, and the database also includes a second index that does not have a time-to-live setting. The method further includes: The number of shards is redefined for the second index; Create a new second index for the second index according to the newly determined number of shards; After migrating the data from the original second index to the new second index, the original second index is deleted. And / or, after creating the first index corresponding to the target period in the database according to the number of shards and the period length, the method further includes: Based on the lifespan and creation time of the first index corresponding to the target period, determine whether the first index corresponding to the target period has expired; In response to the expiration of the first index corresponding to the target period and the fact that the first index corresponding to the target period is not currently being actively queried, the first index corresponding to the target period is asynchronously cleaned up; Wherein, the thread used to clean up the first index has a lower priority than the thread used to write data to the database; and / or, the cleaning up of the first index corresponding to the target period includes merging multiple expired index execution segments before cleaning; and / or, when the amount of data in the first index corresponding to the target period is greater than a preset amount of data, the asynchronous cleaning up of the first index corresponding to the target period is performed when the cluster load of the database is at a low point.

9. An electronic device, characterized in that, The electronic device includes a processor and a memory, the processor being coupled to the memory, and the processor being configured to execute one or more steps of the database management method according to any one of claims 1 to 8 based on instructions stored in the memory.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that is executed by a processor to implement the steps of the database management method as described in any one of claims 1 to 8.

Images