A data storage and query method, system, device and storage medium

Abstract

The application discloses a data storage and query method, system, device and storage medium, and the method comprises the following steps: storing a to-be-stored index into a first storage area of a database, wherein the database comprises the first storage area and a second storage area, and the access speed and storage cost of the first storage area are higher than those of the second storage area; in the case that a storage data migration condition is met, updating a first storage time length of each stored index in the first storage area and a second storage time length of each stored index in the second storage area based on a target linear programming model; and determining whether to migrate each stored index to the second storage area according to the updated first storage time length and the updated second storage time length, deleting each stored index in the second storage area, and executing. By using the method disclosed by the application, the data query efficiency can be improved on the basis of controlling the storage cost.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and specifically to a data storage and retrieval method, system, device, and storage medium. Background Technology

[0002] In today's digital age, businesses rely on application logs to monitor, analyze, and optimize their operations. These logs are massive, often reaching terabytes (TB) or petabytes (PB). To quickly retrieve and query this data, many businesses have adopted high-performance search engines like Elasticsearch. While Elasticsearch itself is high-performance, the continuous growth of log data and complex query requirements have led to increasingly slower query response times, sometimes exceeding ten seconds, significantly impacting business efficiency and user experience.

[0003] To improve query performance, current technologies typically use high-speed disks, such as SSDs (Solid State Disks), as data storage media. However, the storage cost of high-speed disks is much higher than that of traditional HDDs (Hard Disk Drives), and migrating all data to SSDs would lead to a significant increase in storage costs. Summary of the Invention

[0004] This invention provides a data storage and retrieval method, system, device, and storage medium to solve the technical problem of how to improve data retrieval efficiency while controlling storage costs.

[0005] To address the aforementioned technical problems, in a first aspect, the present invention provides a data storage method, the method comprising:

[0006] The index to be stored is stored in the first storage area of ​​the database, which includes the first storage area and the second storage area. The access speed and storage cost of the first storage area are both higher than those of the second storage area.

[0007] Under the condition of meeting the storage data migration conditions, based on the objective linear programming model, the first storage duration of each existing index in the first storage area and the second storage duration of each existing index in the second storage area are updated respectively. The objective linear programming model is constructed according to the query time, storage cost and migration capability of the database.

[0008] Based on the updated first storage duration and the updated second storage duration, determine whether to migrate each of the existing indexes to the second storage area, and delete each of the existing indexes in the second storage area, and execute the procedure.

[0009] Optionally, the objective linear programming model is constructed with minimizing the query time and storage cost of the database as the objective function and one or more of the following conditions as constraints;

[0010] The following conditions include the upper limit of the total storage time of the existing index, the upper limit of the storage volume of the first storage area, the upper limit of the storage volume of the second storage area, and the upper limit of the migration capability from the first storage area to the second storage area.

[0011] Optionally, the objective function of the target linear programming model is minL(t) i ,w i ),in:

[0012]

[0013] Among them, t i w represents the first required storage duration for the i-th existing index. i Let represent the second required storage duration for the i-th existing index, α represent the preset query time cost coefficient, β represent the preset storage cost coefficient, N represent the set of existing indexes, s(i,k) represent the total storage amount of the i-th existing index in the k-th hour most recently, and f(i,t) represent the storage amount of the i-th existing index in the k-th hour most recently. i ) represents the total number of queries performed on the i-th existing index in the second storage region within the most recent preset number of days, g(i) represents the total number of queries performed on the i-th existing index in both the first and second storage regions within the preset number of days, and C h C represents the unit storage cost of the first storage area. c The unit storage cost of the second storage area is represented by i, where i is a positive integer, k < Q*24, and Q represents the preset number of days.

[0014] Optionally, the constraints of the objective linear programming model are as follows:

[0015]

[0016]

[0017]

[0018]

[0019] Among them, t i w represents the first required storage duration for the i-th existing index. i s(i,k) represents the second required storage duration for the i-th existing index, and s(i,k) represents the total storage amount of the i-th existing index in the k-th hour most recently. i ) indicates that the i-th existing index is in the most recent t-th index.i The total storage amount within hours, where N represents the set of stored indexes, and H... s H represents the maximum storage capacity of the first storage region. c Indicates the maximum storage capacity of the second storage region, IO. max T represents the upper limit of I / O for migrating from the first storage area to the second storage area. max This represents the upper limit of the time required to migrate from the first storage area to the second storage area, where i is a positive integer, k < Q*24, and Q represents the preset number of days.

[0020] Optionally, the constraints of the objective linear programming model are as follows:

[0021]

[0022]

[0023]

[0024]

[0025] Among them, t i t' represents the first required storage duration for the i-th existing index. i w represents the difference between the first required storage time for the i-th existing index and the preset data migration interval Δt. i s(i,k) represents the second required storage duration for the i-th existing index, and s(i,k) represents the total storage amount of the i-th existing index in the k-th hour most recently. i ) indicates that the i-th existing index is in the most recent t-th index. i The total storage amount within hours, where N represents the set of stored indexes, and H... s H represents the maximum storage capacity of the first storage region. c Indicates the maximum storage capacity of the second storage region, IO. max T represents the upper limit of I / O for migrating from the first storage area to the second storage area. max This represents the upper limit of the time required to migrate from the first storage area to the second storage area, where i is a positive integer, k < Q*24, and Q represents the preset number of days.

[0026] Optionally, determining whether to migrate each of the existing indexes to the second storage area based on the updated first storage duration and the updated second storage duration includes:

[0027] For each of the existing indexes, if the updated first storage duration is not greater than the historical first storage duration, then the current existing index is migrated to the second storage area.

[0028] Otherwise, the currently existing index will not be migrated to the second storage area.

[0029] In a second aspect, the present invention provides a data storage system, comprising:

[0030] A storage module is used to store the index to be stored in the first storage area of ​​the database, the database including the first storage area and the second storage area, wherein the access speed and storage cost of the first storage area are higher than those of the second storage area;

[0031] The calculation module is used to update the first storage duration of each existing index in the first storage area and the second storage duration of each existing index in the second storage area based on the objective linear programming model, provided that the storage data migration conditions are met. The objective linear programming model is constructed based on the query time, storage cost and migration capability of the database.

[0032] The migration module is used to determine whether to migrate each of the existing indexes to the second storage area based on the updated first storage duration and the updated second storage duration, and to delete each of the existing indexes in the second storage area, and to execute the migration.

[0033] Thirdly, embodiments of the present invention provide a data query method based on the data storage system of the second aspect, comprising:

[0034] Receive user query requests, the user query requests including the index to be queried and the query time range;

[0035] Based on the first storage duration corresponding to the index to be queried, the query time range is divided into a first time range and a second time range;

[0036] Based on the index to be queried and the first time range, a search is performed in the first storage area, and a first search result is returned.

[0037] Based on the index to be queried and the second time range, a search is performed in the second storage area, the ID of the second search result is cached, and the page number corresponding to the ID is updated on the returned page.

[0038] Fourthly, the present invention provides a computer device, including a memory and a processor, wherein:

[0039] The memory is used to store computer programs;

[0040] The processor is used to read the program in the memory and execute the steps of a data storage method as provided in the first aspect above or a data query method as provided in the third aspect above.

[0041] Fifthly, the present invention provides a computer-readable storage medium having a readable computer program stored thereon, which, when executed by a processor, implements the steps of a data storage method as provided in the first aspect or a data query method as provided in the third aspect.

[0042] Compared with the prior art, the data storage and retrieval method, system, device and storage medium provided by the present invention have the following beneficial effects:

[0043] When storing data, the indexes to be stored are first stored in the first storage area of ​​the database. Then, when the data migration conditions are met, the objective linear programming model is used to update the first required storage time for each existing index in the first storage area and the second required storage time for each existing index in the second storage area. Based on the updated first required storage time, it is determined whether to migrate each existing index in the first storage area to the second storage area, and based on the updated second required storage time, it is determined whether to delete each existing index in the second storage area. Because the objective linear programming model takes into account the impact of database query time, storage cost, and migration capability, and executes the migration of existing indexes in the first storage area to the second storage area, and because the access speed and storage cost of the first storage area are higher than those of the second storage area, query efficiency is improved while storage costs are controlled. Attached Figure Description

[0044] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention, and not all embodiments. For those skilled in the art, other drawings obtained from these drawings without creative effort are all within the scope of protection of this application.

[0045] Figure 1 This is a flowchart of a data storage method provided in an embodiment of the present invention;

[0046] Figure 2 This is a schematic diagram of the structure of a data storage system provided in an embodiment of the present invention;

[0047] Figure 3 This is a flowchart of a data query method provided in an embodiment of the present invention;

[0048] Figure 4 This is a schematic diagram illustrating an application scenario of a data storage and query method provided in an embodiment of the present invention;

[0049] Figure 5 This is a schematic diagram of the structure of a computer device provided in an embodiment of the present invention;

[0050] Figure 6 This is a schematic diagram of the structure of a computer-readable storage medium provided in an embodiment of the present invention. Detailed Implementation

[0051] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0052] To make the description of this disclosure more detailed and complete, illustrative descriptions of embodiments and specific examples of the present invention are provided below; however, these are not the only forms of implementing or utilizing the specific embodiments of the present invention. The embodiments cover features of multiple specific embodiments and the methods, steps, and their order for constructing and operating these specific embodiments. However, other specific embodiments may also be used to achieve the same or equivalent functions and step sequences. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without inventive effort are within the scope of protection of this application.

[0053] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in sequences other than those illustrated or described herein.

[0054] Furthermore, in the description of the embodiments of this application, "multiple" refers to two or more, and other quantifiers are similarly understood. The preferred embodiments described herein are only for illustrating and explaining the present invention and are not intended to limit the present invention. Moreover, in the absence of conflict, the embodiments of this application and the features in the embodiments can be combined with each other.

[0055] To address the high cost of storing all data on high-speed disks in existing technologies, a dynamic hot and cold data storage strategy is developed for Elasticsearch (ES). Based on the query characteristics of each index, this strategy ensures that frequently queried log data is stored on high-speed disks, while data from infrequent queries remains on lower-cost storage media. ES is an open-source, distributed, RESTful search and analytics engine based on Lucene, commonly used for log retrieval, analysis, and other search-related applications.

[0056] Example 1

[0057] like Figure 1 The flowchart described above, which is an embodiment of the present invention, provides a data storage method and includes the following steps.

[0058] Step S101: Store the index to be stored in the first storage area of ​​the database. The database includes the first storage area and the second storage area. The access speed and storage cost of the first storage area are higher than those of the second storage area.

[0059] The process involves retrieving the index to be stored. An index is a collection of similar documents, where each document is a data record with multiple fields, each with its own data type and value. These data records can be generated by the business system during business processing; they can be business data or historical query data, depending on the specific circumstances. This embodiment of the invention does not impose any specific limitations on this.

[0060] The index described above consists of one or more shards, each of which is an independent part of the index data. Shards can be primary shards or replicated shards. Primary shards are responsible for storing the data, while replicated shards provide data redundancy and fault tolerance.

[0061] When creating the index, several parameters can be configured, such as the number of shards, replication strategy, and mapping. Mapping is the index schema definition, which specifies how documents and their fields are stored and indexed.

[0062] In this embodiment of the invention, the database is pre-divided into two regions: a first storage region and a second storage region. The scope of the first and second storage regions can be determined according to actual conditions, and this embodiment of the invention does not impose specific limitations on this. The access speed and storage cost of the first storage region are both higher than those of the second storage region. This can be understood as the query performance of the first storage region being higher than that of the second storage region, but the cost of the first storage region is higher than that of the second storage region. The first storage region can be regarded as a hot storage region, indicating that the data stored in the first storage region is frequently accessed. The second storage region can be regarded as a cold storage region, indicating that the data stored in the second storage region is accessed less frequently.

[0063] When storing an index to be stored, the index to be stored is first stored in the first storage area, and the index to be stored becomes an existing index in the first storage area.

[0064] Step S102: Under the condition of meeting the storage data migration conditions, based on the objective linear programming model, update the first storage duration of each existing index in the first storage area and the second storage duration of each existing index in the second storage area. The objective linear programming model is constructed according to the query time, storage cost, and migration capability of the database. In order to control storage costs while improving data query efficiency, it is necessary to reasonably allocate the number of indexes in the first storage area and the second storage area, storing indexes with higher query frequency in the first storage area and indexes with lower query frequency in the second storage area.

[0065] In this embodiment of the invention, under the condition of satisfying the storage data migration, the first storage duration to be stored for each existing index in the first storage area and the second storage duration to be stored for each existing index in the second storage area are calculated using a target linear programming model.

[0066] In this embodiment of the invention, the storage data migration conditions are the conditions that need to be met before data migration. These storage data migration conditions are preset and can be determined according to actual circumstances. This embodiment of the invention does not impose specific limitations on them. For example, the storage data migration conditions may involve performing migration operations at regular intervals. When the preset migration time is reached, the corresponding migration operation is performed; if the preset migration time has not been reached, the corresponding migration operation is not performed.

[0067] Since query time, storage cost, and the impact of data migration need to be considered when performing data migration operations, the objective linear programming model in this embodiment of the invention is constructed based on the database's query time, storage cost, and migration capability. The objective linear programming model updates the first and second storage durations of each existing index, so that the query time, storage cost, and migration capability corresponding to the first and second storage durations are all within an acceptable range, achieving a balance.

[0068] As an optional implementation, the objective linear programming model is constructed with minimizing the query time and storage cost of the database as the objective function and one or more of the following conditions as constraints;

[0069] The following conditions include the upper limit of the total storage time of the existing index, the upper limit of the storage volume of the first storage area, the upper limit of the storage volume of the second storage area, and the upper limit of the migration capability from the first storage area to the second storage area.

[0070] The objective linear programming model is a linear programming model. The three variables of a linear programming model are decision variables, objective function, and constraints. In this embodiment of the invention, the decision variables of the objective linear programming model are the first storage duration and the second storage duration. The objective function is to minimize the query time and storage cost of the database. The constraints are one or more of the following: the upper limit of the total storage duration of the existing indexes, the upper limit of the storage volume of the first storage area, the upper limit of the storage volume of the second storage area, and the upper limit of the migration capability from the first storage area to the second storage area. The specific constraints can be determined according to the actual situation, and this embodiment of the invention does not impose specific limitations on them.

[0071] It should be noted that the upper limit of the total storage time for existing indexes represents the total storage time for the first and second existing indexes after the update; the upper limit of the storage capacity of the first storage area refers to the maximum number of indexes that the first storage area can store; the upper limit of the storage capacity of the second storage area refers to the maximum number of indexes that the second storage area can store; and the upper limit of the migration capacity from the first storage area to the second storage area refers to the maximum number of indexes that can be migrated from the first storage area to the second storage area.

[0072] The specific calculation formulas for the above objective function and constraints can be determined according to the actual situation, and the embodiments of the present invention do not impose specific limitations on them.

[0073] As an optional implementation, the objective function of the target linear programming model is minL(t) i ,w i ),in:

[0074]

[0075] Among them, t i w represents the first required storage duration for the i-th existing index. i Let represent the second required storage duration for the i-th existing index, α represent the preset query time cost coefficient, β represent the preset storage cost coefficient, N represent the set of existing indexes, s(i,k) represent the total storage amount of the i-th existing index in the k-th hour most recently, and f(i,t) represent the storage amount of the i-th existing index in the k-th hour most recently. i ) represents the total number of queries performed on the i-th existing index in the second storage region within the most recent preset number of days, g(i) represents the total number of queries performed on the i-th existing index in both the first and second storage regions within the preset number of days, and C h C represents the unit storage cost of the first storage area. c The unit storage cost of the second storage area is represented by i, where i is a positive integer, k < Q*24, and Q represents the preset number of days.

[0076] In the first storage area, the first required storage duration must be calculated for each existing index. The first required storage duration for each existing index can be the same or different. Similarly, in the second storage area, the second required storage duration must be calculated for each existing index. The second required storage duration for each existing index can be the same or different.

[0077] It should be noted that Q in formula (1) represents the preset number of days, the size of which can be determined according to the actual situation. This embodiment of the invention does not impose a specific limitation on this. In this embodiment of the invention, Q is 7.

[0078] As an optional implementation, the constraints of the objective linear programming model are as follows:

[0079]

[0080]

[0081]

[0082]

[0083] Among them, t i w represents the first required storage duration for the i-th existing index. i s(i,k) represents the second required storage duration for the i-th existing index, and s(i,k) represents the total storage amount of the i-th existing index in the k-th hour most recently. i ) indicates that the i-th existing index is in the most recent t-th index. i The total storage amount within hours, where N represents the set of stored indexes, and H... s H represents the maximum storage capacity of the first storage region. c Indicates the maximum storage capacity of the second storage region, IO. max T represents the upper limit of I / O for migrating from the first storage area to the second storage area. max This represents the upper limit of the time required to migrate from the first storage area to the second storage area, where i is a positive integer, k < Q*24, and Q represents the preset number of days.

[0084] In the above formulas (1) and (2), f(i,t) iThe expression represents the total number of queries performed on the i-th existing index in the second storage area within the most recent preset number of days. For a query request on the i-th existing index, the query time and the first and second required storage durations of the i-th existing index are used to determine whether the query should be performed in the first or second storage area. Specifically, if the query time is less than the first required storage duration, the query time can fall entirely within the first storage area, and the query is performed there. If the query time is greater than the first required storage duration, the query time is divided into two parts: first, the first required storage duration is queried in the first storage area, and then the remaining duration is queried in the second storage area.

[0085] For example, let's take the total number of queries to the second storage area of ​​the i-th existing index in the last 7 days as an example. In the last 7 days, a query request occurred at 16:00:00 on 2023-10-03, and the query range was [12:00:00 on 2023-10-03, 15:00:00]. If the first required storage time t for the i-th existing index... i If the storage time is 5 hours, then the request falls entirely within the first storage area, excluding the second storage area, and this query request is not counted. If the first storage duration t of the i-th existing index is... i If the value is 2, then the query request needs to be queried not only in the first storage area but also in the second storage area. This query request count is then used to summarize the total number of queries within 7 days.

[0086] g(i) represents the total number of queries performed on the i-th existing index within the preset number of days in the first and second storage regions. Query requests fall into two categories: one where the query only occurs in the first storage region, and the other where the query request is divided into two sub-queries, one in the first storage region and the other in the second storage region. Regardless of whether the query occurs in the first or second storage region, it is counted as part of the total number of queries performed, which is the total number of queries for the query request. f(i,t) i This refers to the number of times the data needs to be retrieved from the second storage area.

[0087] As another optional implementation, the constraints of the objective linear programming model are as follows:

[0088]

[0089]

[0090]

[0091]

[0092] Among them, t i t' represents the first required storage duration for the i-th existing index. i w represents the difference between the first required storage duration of the i-th existing index and the preset data migration interval Δt. i s(i,k) represents the second required storage duration for the i-th existing index, and s(i,k) represents the total storage amount of the i-th existing index in the k-th hour most recently. i ) indicates that the i-th existing index is in the most recent t-th index. i The total storage amount within hours, where N represents the set of stored indexes, and H... s H represents the maximum storage capacity of the first storage region. c Indicates the maximum storage capacity of the second storage region, IO. max T represents the upper limit of I / O for migrating from the first storage area to the second storage area. max This represents the upper limit of the time required to migrate from the first storage area to the second storage area, where i is a positive integer, k < Q*24, and Q represents the preset number of days.

[0093] Considering that the storage size of each existing index fluctuates regularly at different hours of the day, although the storage size of each row remains unchanged, the number of logs will increase significantly during certain periods due to concentrated business volume, resulting in a significant increase in the total storage for that hour. Therefore, the original constraints (Formula 2) can be further improved.

[0094] If a large number of existing indexes are migrated simultaneously, it may lead to excessive I / O. That is, the I / O portion of the constraint can be based on Δt = t. i -t' i Calculate the migration storage size That is, change the last expression in the original constraint formula (2) to: The new constraint condition can then be obtained as formula (3).

[0095] Continue with the above Figure 1 As explained in step S103, based on the updated first storage duration and the updated second storage duration, it is determined whether to migrate each of the existing indexes to the second storage area, and to delete each of the existing indexes in the second storage area, and then the process is executed.

[0096] After calculating the first storage duration for each existing index in the first storage area and the second storage duration for each index in the second storage area, each existing index is judged according to a preset time interval. In each judgment, each existing index has one opportunity to be eliminated from the first storage area and moved to the second storage area, or to be deleted from the second storage area.

[0097] As an optional implementation, determining whether to migrate each of the existing indexes to the second storage area based on the updated first storage duration and the updated second storage duration includes:

[0098] For each of the existing indexes, if the updated first storage duration is not greater than the historical first storage duration, then the current existing index is migrated to the second storage area.

[0099] Otherwise, the currently existing index will not be migrated to the second storage area.

[0100] For example, by combining the updated first storage duration and the updated second storage duration, it is determined whether to migrate each audited existing index to the aforementioned second storage area and delete each existing index in the aforementioned second storage area. After that, the data migration can be carried out using the double method or the midpoint method.

[0101] Data migration decisions are made based on the updated first and second required storage durations. For each existing index in the first storage area, an updated first required storage duration is generated. If the current first required storage duration is greater than the previous one, the first storage area will not evict the existing index when the eviction timing arrives, until the existing index's storage duration in the first storage area grows to the updated first required storage duration over time. If the updated first required storage duration is less than the previous one, the first storage area will evict several hours of existing indexes to the second storage area when the eviction timing arrives.

[0102] Similarly, if the updated second required storage duration is longer than the previous one, then when the eviction period arrives, the second storage area will not delete the existing index until the existing index's storage duration in the second storage area grows to the updated second required storage duration over time; if the updated second required storage duration is shorter than the previous one, then when the eviction period arrives, the second storage area will delete the existing index for several hours.

[0103] When replacing stored indexes that have been stored for several hours in the first storage area with those in the second storage area, and when deleting stored indexes that have been stored for several hours in the second storage area, for the double method, the aforementioned number of hours is twice the preset time interval. For example, if the preset time interval for judging each stored index is 1 hour, then stored indexes that have been stored for 2 hours in the first storage area are replaced with those in the second storage area, and stored indexes that have been stored for 2 hours are deleted in the second storage area. For the midpoint method, the aforementioned number of hours is the average of the first required storage time for each stored index update and the historical first required storage time. For example, if the first required storage time for a certain stored index update is 2 hours and the historical first required storage time is 1 hour, and their average is 1.5 hours, then stored indexes that have been stored for 1.5 hours in the first storage area are replaced with those in the second storage area, and stored indexes that have been stored for 1.5 hours are deleted in the second storage area.

[0104] This invention proposes a data storage method. During data storage, the indexes to be stored are first stored in the first storage area of ​​the database. Then, when data migration conditions are met, a target linear programming model is used to update the first required storage duration of each existing index in the first storage area and the second required storage duration of each existing index in the second storage area. Based on the updated first required storage duration, it is determined whether to migrate each existing index in the first storage area to the second storage area, and based on the updated second required storage duration, it is determined whether to delete each existing index in the second storage area. Because the target linear programming model considers the impact of database query time, storage cost, and migration capability, and executes the migration of existing indexes from the first storage area to the second storage area, and because the access speed and storage cost of the first storage area are higher than those of the second storage area, this method improves query efficiency while controlling storage costs.

[0105] Example 2

[0106] Based on the above data storage method, embodiments of the present invention provide a data storage system, the structural diagram of which is shown below. Figure 2 As shown, the system includes a storage module 210, a computing module 220, and a migration module 230, wherein:

[0107] Storage module 210 is used to store the index to be stored in the first storage area of ​​the database, the database including the first storage area and the second storage area, the access speed and storage cost of the first storage area are higher than those of the second storage area;

[0108] The calculation module 220 is used to update the first storage duration of each existing index in the first storage area and the second storage duration of each existing index in the second storage area based on the target linear programming model, provided that the storage data migration conditions are met. The target linear programming model is constructed based on the query time, storage cost and migration capability of the database.

[0109] The migration module 230 is used to determine whether to migrate each of the existing indexes to the second storage area based on the updated first storage duration and the updated second storage duration, and to delete each of the existing indexes in the second storage area, and to execute the migration.

[0110] For other details regarding the implementation of the above technical solution by each module in the above data storage system, please refer to the description in the data storage method provided in the above embodiments of the invention, which will not be repeated here.

[0111] Each module in the aforementioned data storage system can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of the computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0112] Example 3

[0113] Based on the aforementioned data storage system, this invention provides a data query method, the flowchart of which is shown below. Figure 3 As shown, the query method includes:

[0114] S310, Receive a user query request, the user query request including the index to be queried and the query time range;

[0115] S320, based on the first storage duration corresponding to the index to be queried, the query time range is divided into a first time range and a second time range;

[0116] S330, based on the index to be queried and the first time range, perform a search in the first storage area and return the first search result;

[0117] S340, based on the index to be queried and the second time range, perform a search in the second storage area, cache the ID of the second search result, and update the page number corresponding to the ID on the returned page.

[0118] In this embodiment of the invention, when performing a data query, a user query request is first received. This user query request includes the index to be queried and the query time range. The index to be queried is the index that needs to be queried. For example, for a user query request for the i-th existing index, based on the query time range of the query request and the first and second storage durations of the i-th existing index, it is determined whether the i-th existing index should be queried in the first storage area or the second storage area.

[0119] Specifically, if the query time falls entirely within the first storage area, the query is performed in the first storage area; otherwise, the query time is divided into two parts, namely the first time range and the second time range. First, the first time range is queried in the first storage area to obtain the first search result, and the first search result is returned to the user; then, the second time range is queried in the second storage area to obtain the ID of the second search result, and the page number corresponding to the ID is updated on the returned page. The front end can also be notified to update the total number of query records and the total number of pages.

[0120] For example, upon receiving a user query request, the system divides the query time range into two parts based on the index to be queried and the corresponding query time range in the user query request, and then distributes them to the corresponding storage areas. For instance, the user query request might be: {"indices":"waybill_net","start_time":"2023-10-01"}.

[0121] Given the current time is 2023-10-03 15:00:00, and the first required storage duration for the index "waybill_net" is 7 hours, the user's query request is split into two requests: {"indices":"waybill_net","start_time":"2023-10-03 15:00:00"}, and the index "waybill_net" is to be queried, the user's query request is split into two requests: {"indices":"waybill_net","start_time":"2023-10-03 15:00:00"}.

[0122] 08:00:00”,”end_time”:”2023-10-0315:00:00”} and

[0123] {"indices":"waybill_net","start_time":"2023-10-01

[0124] 00:00:00”,”end_time”:”2023-10-0308:00:00”}. The first request queries the first storage area, and the second request queries the second storage area.

[0125] In this embodiment of the invention, page numbers on the user's page are mapped to query results, allowing users to access query records through a specified page without the need for the backend to cache a large number of intermediate results.

[0126] This invention also provides an application scenario diagram of a data storage and query method, such as... Figure 4 As shown, the above application scenarios include business systems, storage management systems, and log query systems. The business system is mainly responsible for internal business processing and generates log data during the processing. The log data is written into the database in the form of an index. The business system is the data producer and only needs to write data to the database, which is usually an Elasticsearch database.

[0127] The log query system is a data consumer. It receives user query requests and then, using the data query method described above, divides the time range in the user query request into two parts based on the index to be queried and the time range corresponding to the index. The parts are then distributed to the first storage area and the second storage area of ​​the database. The system performs searches in the first storage area and the second storage area respectively, and returns the search results from the first storage area while caching the index ID of the second storage area.

[0128] After the business system writes the index to be stored into the database, the storage management system executes the above data storage method, and is responsible for generating and adjusting the storage policy. The storage policy calculation service collects data and calculates the first storage duration to be stored for each existing index in the first storage area and the second storage duration to be stored for the corresponding second storage area. The migration execution service guides the first storage area and the second storage area in the database to perform migration operations based on the first storage duration and the second storage duration.

[0129] The storage management system acts as the data manager, periodically analyzing the first and second retention periods of each stored index based on the query system's historical records and database metadata, and periodically performing data migration operations.

[0130] Based on the above data storage methods or data query methods, such as Figure 5 As shown in the diagram, this embodiment of the invention also provides a schematic diagram of a computer device. The identification device includes a processor 51 and a memory 52 coupled to the processor 51. The memory 52 stores a computer program. When the computer program is executed by the processor 51, the processor 51 performs the steps of the data storage method or data query method described in the above embodiments.

[0131] For other details regarding the implementation of the above technical solution by the processor 51 in the above computer device, please refer to the description in the data storage method or data query method provided in the above embodiments of the invention, which will not be repeated here.

[0132] The processor 51 can also be called a CPU (Central Processing Unit). The processor 51 may be an integrated circuit chip with signal processing capabilities. The processor 51 can also be a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The general-purpose processor can be a microprocessor, or the processor 51 can be any conventional processor.

[0133] like Figure 6 As shown in the diagram, this embodiment of the invention also provides a schematic diagram of a computer-readable storage medium, on which a readable computer program 61 is stored. The computer program 61 can be stored in the storage medium in the form of a software product, including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods described in various embodiments of the invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, magnetic disks or optical disks, ROM (Read-Only Memory), RAM (Random Access Memory), or terminal devices such as computers, servers, mobile phones, and tablets.

[0134] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, apparatuses, or modules, and may be electrical, mechanical, or other forms.

[0135] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0136] Furthermore, the functional modules in the various embodiments of this application can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can be stored in a computer-readable storage medium.

[0137] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can store or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0138] The technical solutions provided in this application have been described in detail above. Specific examples have been used in this application to illustrate the principles and implementation methods of this application. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

[0139] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0140] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or

[0141] or box Figure 1 A device that provides the functions specified in one or more boxes.

[0142] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0143] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0144] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A data storage method, characterized in that, include: The index to be stored is stored in the first storage area of ​​the database, which includes the first storage area and the second storage area. The access speed and storage cost of the first storage area are both higher than those of the second storage area. Under the condition of meeting the storage data migration conditions, based on the objective linear programming model, the first storage duration of each existing index in the first storage area and the second storage duration of each existing index in the second storage area are updated respectively. The objective linear programming model is constructed according to the query time, storage cost and migration capability of the database. Based on the updated first storage duration and the updated second storage duration, determine whether to migrate each of the existing indexes to the second storage area, and delete each of the existing indexes in the second storage area, and execute the above steps. The objective function of the target linear programming model is: ,in: ， in, Indicates the first The first storage duration for an existing index. Indicates the first The second storage duration for an existing index. This represents the cost coefficient for the preset query time. This represents the preset storage cost coefficient. This represents the set of existing indices. Indicates the first The existing index is in the most recent... Total storage volume per hour Indicates the first The total number of queries performed on an existing index in the second storage area within the most recent preset number of days. Indicates the first The total number of queries performed on the first and second storage regions within the preset number of days for the existing index. This represents the unit storage cost of the first storage area. This represents the unit storage cost of the second storage area. It is a positive integer. , This indicates the preset number of days; The constraints of the objective linear programming model are as follows: ， ， ， ， in, Indicates the first The first storage duration and preset data migration interval for each existing index. The difference, This represents the set of existing indices. This indicates the maximum storage capacity of the first storage region. This indicates the maximum storage capacity of the second storage area. This indicates the upper limit of I / O for migrating from the first storage area to the second storage area. This indicates the maximum time required to migrate from the first storage area to the second storage area.

2. The data storage method according to claim 1, characterized in that, The objective linear programming model is constructed with minimizing the query time and storage cost of the database as the objective function and one or more of the following conditions as constraints; The following conditions include the upper limit of the total storage time of the existing index, the upper limit of the storage volume of the first storage area, the upper limit of the storage volume of the second storage area, and the upper limit of the migration capability from the first storage area to the second storage area.

3. The data storage method according to claim 1 or 2, characterized in that, The constraints of the objective linear programming model are also given by the following formula: ， ， ， ， in, Indicates the first The first storage duration for an existing index. Indicates the first The second storage duration for an existing index. Indicates the first The existing index is in the most recent... Total storage volume per hour Indicates the first The existing index is in the most recent... Total storage volume per hour This represents the set of existing indices. This indicates the maximum storage capacity of the first storage region. This indicates the maximum storage capacity of the second storage area. This indicates the upper limit of I / O for migrating from the first storage area to the second storage area. This indicates the maximum time required to migrate from the first storage area to the second storage area. It is a positive integer. , This indicates the preset number of days.

4. The data storage method according to claim 1, characterized in that, The step of determining whether to migrate each of the existing indexes to the second storage area based on the updated first storage duration and the updated second storage duration includes: For each of the existing indexes, if the updated first storage duration is not greater than the historical first storage duration, then the current existing index is migrated to the second storage area. Otherwise, the currently existing index will not be migrated to the second storage area.

5. A data storage system, said system being used to implement the steps of the data storage method as described in any one of claims 1-4, characterized in that, include: A storage module is used to store the index to be stored in the first storage area of ​​the database, the database including the first storage area and the second storage area, wherein the access speed and storage cost of the first storage area are higher than those of the second storage area; The calculation module is used to update the first storage duration of each existing index in the first storage area and the second storage duration of each existing index in the second storage area based on the objective linear programming model, provided that the storage data migration conditions are met. The objective linear programming model is constructed based on the query time, storage cost and migration capability of the database. The migration module is used to determine whether to migrate each of the existing indexes to the second storage area based on the updated first storage duration and the updated second storage duration, and to delete each of the existing indexes in the second storage area, and to execute the migration.

6. A data query method based on the data storage system of claim 5, characterized in that, include: Receive user query requests, the user query requests including the index to be queried and the query time range; Based on the first storage duration corresponding to the index to be queried, the query time range is divided into a first time range and a second time range; Based on the index to be queried and the first time range, a search is performed in the first storage area, and a first search result is returned. Based on the index to be queried and the second time range, a search is performed in the second storage area, the ID of the second search result is cached, and the page number corresponding to the ID is updated on the returned page.

7. A computer device, characterized in that, Includes memory and processor, wherein: The memory is used to store computer programs; The processor is used to read the computer program in the memory and execute the steps of the data storage method as described in any one of claims 1 to 4 or the data query method as described in claim 6.

8. A computer-readable storage medium, characterized in that, It stores a readable computer program that, when executed by a processor, implements the steps of the data storage method as described in any one of claims 1 to 4 or the data query method as described in claim 6.

Images