Mongo query optimization method based on time sequence prediction and generation

Through the method based on time series prediction and generation, analyzing historical query records to generate popularity values, predicting and executing query tasks, the problem of slow query speed of Mongo database complex aggregation analysis is solved, and efficient data query response and cache hit rate improvement is achieved. , avoiding the increase in hardware costs.

CN118964402APending Publication Date: 2024-11-15HUAXIN CONSULTATING CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410970717.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-19
Publication Date
2024-11-15

AI Technical Summary

Technical Problem

Mongo database is slow in querying in complex aggregation analysis. The existing technology requires data to be moved to OLAP database, resulting in increased hardware costs and extended data flow, which cannot effectively improve query response speed.

Method used

Using a query optimization method based on time series prediction and generation formula, the popularity value is generated by analyzing historical query records, predicting query tasks, and performing query tasks in different time intervals regularly and caches results, improving the response speed of data query and avoiding manual operation and maintenance.

Benefits of technology

It improves the response speed of Mongo database data query under large data volume, improves the hit rate of cached results, and combines the Mongo database's own characteristics without increasing the cost of software and hardware, avoiding the hardware cost and complexity brought about by data migration.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118964402A_ABST
    Figure CN118964402A_ABST
Patent Text Reader

Abstract

The invention discloses a Mongo query optimization method based on time sequence prediction and a generative formula. The problem that an existing Mongo database cannot effectively improve the query speed in the aspect of complex aggregation analysis is solved. The method comprises the following steps: acquiring a query request by a database process, performing aggregation analysis judgment, performing cache hit judgment on the query request related to aggregation analysis, acquiring a materialized view function for a miss situation, and acquiring a query result according to a function execution result; the cache result comprises an execution result of regularly executing the query task, and the query task is constructed by performing popularity prediction on each time period according to a historical query record and obtaining a view function with high popularity. According to the method, the query task is formulated according to the heat value, the cache result is executed and updated regularly, the net weight rate of the cache result is increased, the response speed of data query is increased, and manual data processing script operation and maintenance are avoided. And in combination with the characteristics of the Mongo database, the software and hardware cost does not need to be increased on the basis of a function prediction and generation mode.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of database management, and in particular to a Mongo query optimization method based on time series prediction and generation. Background Art

[0002] As a semi-structured database, Mongo's query and analysis capabilities are weaker than those of OLAP databases, especially in complex aggregate analysis where indexes cannot effectively improve query speed. Therefore, in related fields, the common approach is to migrate Mongo data to OLAP databases for data analysis and data mining. However, if data migration is introduced, more components will be introduced due to the extension of the data flow path, and the amount of code development will increase. At the same time, due to the redundant storage of data, more hardware costs will be incurred. Therefore, it is necessary to design a method that combines the Mongo database's own image and does not increase the cost of software and hardware to improve the data query response speed of the Mongo database under large data volumes. Summary of the invention

[0003] The present invention mainly solves the problem in the prior art that the Mongo database cannot effectively improve the query speed in complex aggregation analysis, and provides a Mongo query optimization method based on time series prediction and generation.

[0004] The above technical problem of the present invention is mainly solved by the following technical solution: a Mongo query optimization method based on time series prediction and generation, characterized in that: it includes: The database process obtains the query request, performs aggregate analysis and judgment, parses the query request involving aggregate analysis, makes a hit judgment based on the cache result, obtains the view function for the miss case, and obtains the query result based on the view function execution result; The cached results include the execution results of scheduled query tasks. The query tasks are constructed by predicting the popularity of each time period based on historical query records and obtaining the most popular view functions.

[0005] The present invention can generate a heat value according to historical query records, make predictions according to the heat value, formulate query tasks according to the prediction results, execute different query tasks in different time intervals according to the query task list and cache the results, thereby improving the response speed of data query and avoiding manual data processing script operation and maintenance. The query task is generated according to the historical query record, and the larger the historical query record data, the more accurate the prediction. The method of the present invention combines the characteristics of the Mongo database itself, based on the prediction and generation function method, without increasing the cost of software and hardware.

[0006] The database process is used to perform related operations on the database. The database process includes a query processor, a query executor, and a query queue. The client initiates a query request to the database, that is, sends a query request to the database process, and the query processor receives the query request for processing.

[0007] As a preferred solution, the heat prediction includes: At a fixed time every day, based on historical query records, count the number of requests per view function per unit time, the median average of query response time, the number of query users and IP addresses, assign weights to each, and sum them up to obtain the heat value of each time interval and build a heat value set; The heat value results are used as sample data and input into the time series prediction model to predict the heat value of each view function in each time interval in the next 24 hours and establish a set of prediction results.

[0008] This solution generates a heat value based on historical query records, and then predicts the heat value through a time series prediction model. A query task list is formulated based on the prediction results, and then the query tasks are executed. The query task is established by selecting prediction results with high heat values, making the execution results easier to use directly, thereby improving the data query response speed.

[0009] A query record is inserted after each query request is executed. The query record includes query statement ID, insertion time, response time, corresponding view function, etc. According to the query record, the view function corresponding to the query, as well as the number of times, number of users, and other information can be obtained. The statistical cycle of historical query records is generally one day, one day is divided into 24 hours, and the unit time is one hour. By analyzing the historical query records, the number of view function requests per unit time, the median average of the query response time, the number of query users and IP addresses, and other data can be calculated.

[0010] As a preferred solution, a hot query statement is obtained to construct a query task, which specifically includes: Read all the prediction result sets in the current time interval, sort the view functions in descending order according to the heat value, obtain the view functions with the heat value in the top X%, and establish a query task.

[0011] Since the query task is executed periodically, that is, the query task of the current time interval is obtained and executed every unit time, the query task is generated by obtaining the view function with the heat value in the top X% and the value of X% can be adjusted according to the query cache hit rate.

[0012] As a preferred solution, the query task is executed regularly, including: Get the query task of the current time interval every unit time, execute the view function corresponding to the query task, and write the function execution result into the aggregate result cache collection.

[0013] The query executor of this solution executes tasks at a scheduled time, executes different query tasks in different time periods and caches the execution results to meet the data query response speed.

[0014] As a preferred solution, the polymer analysis and judgment includes: Obtain a query request, and evaluate whether aggregation analysis is involved based on the query request; If no aggregate analysis is involved, the query request is forwarded to the database, and the database returns the query result; If aggregation analysis is involved, perform cache result hit judgment.

[0015] After obtaining the query request, this solution first performs aggregation analysis and judgment. When aggregation analysis is involved, the corresponding aggregation result cache set is read for hit judgment. If aggregation analysis is not involved, it is directly sent to the database for query and the query result is returned.

[0016] As a preferred solution, parsing the query request includes: The database process obtains the query request, extracts the query statement, obtains the query target collection name according to the query statement, calculates the collection status code, obtains the aggregate result cache collection name according to the target collection name, extracts the aggregate statement of the query request, and performs hashing to obtain the query code.

[0017] In this solution, the query processor QueryHandler of the database process MongoObserver receives the query request, extracts the query statement according to the query request to obtain the query target collection name CollName, calculates the collection status code CollStats, obtains the aggregation result cache collection name CacheColName according to the target collection name, extracts the aggregation statement of the query request and hashes it to obtain the query code QueryCode.

[0018] As a preferred solution, hit judgment is performed according to the cache structure, including: According to the query code and the query target collection status code, query the aggregate cache result in the collection corresponding to the aggregate result cache collection name to determine whether there is a cache result with the same query code; If there is a cached result, determine whether the collection status code is the same. If so, confirm that the cached result is the latest query result and return the query result. If not, the cached result is not hit and proceed to the step of obtaining the view function. If there is no cached result, the cached result is not hit, and the view function acquisition step is performed.

[0019] In this solution, the query processor QueryHandler of the database process MongoObserver searches the Mongo database for the collection corresponding to the name of the aggregated result cache collection, searches for the cached result with the same query code, and compares the collection status codes when a cached result is found. After the comparison passes, it is confirmed that the cached result is the latest query result. The database process MongoObserver obtains the query result and returns the query result to the client.

[0020] As a preferred solution, obtaining the view function includes: Use the query code and query target collection name to check whether there is a corresponding view function in the view function list collection. If it exists, write the view function into the query queue. If it does not exist, create a new view function and write the view function into the query queue.

[0021] In this solution, the query processor QueryHandler uses the query code QueryCode and the query target collection name CollName to query whether there is a view function with the same query code and query target collection name in the view function list collection QueryViewFuncList. If it exists, the view function is written into the query queue QueryQueue, and the query executor QueryExecutor is awakened to execute the function in the query queue. Among them, the database process MongoObserver includes the query processor QueryHandler, the query executor QueryExecutor and the query queue QueryQueue. The query queue QueryQueue supports data persistence in the database to ensure the fault recovery mechanism; if it does not exist, a new view function is created, and then the view function is written into the query queue QueryQueue, and the query executor QueryExecutor is awakened to execute the function in the query queue. After the query executor QueryExecutor executes the view function, it returns the execution result to the query processor QueryHandler, and the query processor QueryHandler returns the query result to the client.

[0022] As a preferred solution, after obtaining the query results and completing a query, the query record is inserted into the query record statistics table.

[0023] Each time a query result is obtained, the query record is inserted into the query statistics table to form a historical query record, which is used for later popularity prediction.

[0024] As a preferred solution, the view function execution includes looping and executing the view function in the query queue until the query queue is empty, executing the view function to obtain the execution result, writing it into the aggregate structure cache collection, and returning the query result.

[0025] The view function is executed by the query executor QueryExecutor. This solution is the immediate execution working mode of the query executor QueryExecutor. The immediate execution mode is that after the query executor QueryExecutor is awakened by the query processor QueryHandler, it cyclically executes the view function in the query queue QueryQueue until the query queue is empty.

[0026] Therefore, the advantages of the present invention are: 1. It can generate heat values ​​based on historical query records, make predictions based on heat values, formulate query tasks based on prediction results, execute different query tasks in different time intervals according to the query task list and cache the results, which improves the response speed of data query and avoids manual data processing script operation and maintenance.

[0027] 2. Generate query tasks based on historical query records. The larger the sample data of historical query records, the more accurate the prediction results, which improves the cache result hit rate and the response speed of data query.

[0028] 3. Combining the characteristics of the Mongo database itself, based on the prediction and generation function method, the data query response speed of the Mongo database under large data volumes is improved, without the need for data migration and without increasing software and hardware costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 It is a schematic diagram of an execution flow of the present invention for determining that no aggregation analysis is involved.

[0030] Figure 2 It is a schematic diagram of an execution flow of determining a cache hit result in the present invention.

[0031] Figure 3 It is a schematic diagram of an execution flow of determining a cache miss result in the present invention.

[0032] Figure 4 It is a schematic diagram of an execution flow of the immediately executed function in the present invention.

[0033] Figure 5 It is a schematic diagram of an execution flow of the timed execution of query tasks in the present invention. DETAILED DESCRIPTION

[0034] The technical solution of the present invention is further specifically described below through embodiments and in conjunction with the accompanying drawings.

[0035] Embodiment 1: This embodiment provides a Mongo query optimization method based on time series prediction and generation, including the following steps: S1. The database process obtains the query request and performs aggregate analysis and judgment; S2. Parse the query request involving aggregate analysis and make a hit judgment based on the cached results; The cached results include the results of scheduled query tasks. The query tasks are constructed by predicting the popularity of each time period based on historical query records and obtaining the most popular view functions. S3. For a miss, obtain the view function and obtain the query result according to the view function execution result.

[0036] The present invention can generate a heat value according to historical query records, make predictions according to the heat value, formulate query tasks according to the prediction results, execute different query tasks in different time intervals according to the query task list and cache the results, thereby improving the response speed of data query and avoiding manual data processing script operation and maintenance. The query task is generated according to the historical query record, and the larger the historical query record data, the more accurate the prediction. The method of the present invention combines the characteristics of the Mongo database itself, based on the prediction and generation function method, without increasing the cost of software and hardware.

[0037] The Mongo database has a database process MongObserver, which is used to perform related operations on the Mongo database. The database process MongObserver includes a query processor QueryHandler, a query executor QueryExecutor, and a query queue QueryQueue. The above units are applied in subsequent processes involving aggregation analysis and judgment and cache result hit judgment.

[0038] As a preferred solution of this embodiment, step S1 specifically includes the following process: S11. The client Client queries the Mongo proxy service, that is, sends a query request to the database process MongoObserver.

[0039] S12. After the database process MongoObserver receives the query request, its internal query processor QueryHandler evaluates the query request to determine whether the query request involves aggregation analysis.

[0040] S13. If no aggregation analysis is involved, e.g. Figure 1As shown, the query processor QueryHandler forwards the query request to the database, the database performs the query and returns the query result to the query processor QueryHandler, and the query processor QueryHandler returns the query result to the client.

[0041] S14. If aggregation analysis is involved, go to step S2.

[0042] After obtaining the query request, the first step is to determine whether it involves aggregation analysis. If it involves aggregation analysis, it will be processed separately. If it involves aggregation analysis, the corresponding aggregation result cache set will be read to make a cache hit judgment. If it does not involve aggregation analysis, it will be directly sent to the database for query and the query results will be returned, which effectively improves the query speed for complex aggregation analysis in the query.

[0043] As a preferred solution of this embodiment, step S2 specifically includes the following process: S21. The query processor QueryHandler extracts the query statement from the query request, obtains the query target collection name CollName according to the query statement, and calculates the collection status code CollStats, obtains the aggregate result cache collection name CacheColName according to the target collection name CollName, extracts the aggregate statement from the query request, and performs hashing to obtain the query code QueryCode.

[0044] S22. The query processor QueryHandler queries the aggregate cache result in the collection corresponding to the aggregate result cache collection name according to the query code QueryCode and the query target collection status code CollStats, and determines whether there is a cache result with the same query code.

[0045] S23. Figure 2 As shown, if there is a cached result, determine whether the collection status code is the same. If the same, confirm that the cached result is the latest query result, insert the query record into the query statistics table, return the query result to the database process MongoObserver, and the database process MongoObserver returns the query result to the client; if they are not the same, the cached result is not hit, and enter step S3. When each query obtains a query result, the query record is inserted into the query statistics table to form a historical query record, which is used for later heat prediction.

[0046] S24. If there is no cache result, the cache result is not hit, and the process goes to step S3.

[0047] By judging the hit of cached results, the hit cached results are directly returned as query results. By reusing the cached results, there is no need to execute queries from the original database, which effectively improves the data query response speed under large data volumes.

[0048] As a preferred solution of this embodiment, Figure 3 As shown, step S3 specifically includes the following process: S31. The query processor QueryHandler checks whether there is a view function with the same query code and query target collection name in the view function list set QueryViewFuncList by using the query code QueryCode and the query target collection name CollName.

[0049] S32. If the view function exists, write the view function into the query queue QueryQueue and go to step S34.

[0050] S33. If the view function does not exist, create a new view function, and then write the view function into the query queue QueryQueue.

[0051] S34. The query processor QueryHandler wakes up the query executor QueryExecutor to execute the view function in the query queue, wherein the query queue QueryQueue supports data persistence in the database to ensure a fault recovery mechanism.

[0052] S35. After executing the view function, the query executor QueryExecutor returns the execution result to the query processor QueryHandler, which returns the query result to the client Client and inserts the query record of the query request into the query statistics table. When each query obtains the query result, the query record is inserted into the query statistics table to form a historical query record, which is used for later heat prediction.

[0053] As a preferred solution of this embodiment, the query executor QueryExecutor includes a scheduled execution working mode, which executes query tasks regularly and writes them into the aggregated result cache collection. The query task is constructed by predicting the popularity of each time period based on historical query records and obtaining the view function with high popularity. Specifically, Figure 5 As shown, the timing execution includes the following processes: S3511. At a fixed time every day, based on historical query records, count the number of requests C per unit time for each view function, the median average of the query response time R, and the number of query users and IPs U, and assign weights λ, β, and α respectively and sum them up to obtain the heat value H of each view function in each time interval, H=λC+βR+αU, and construct a heat value set. The cycle of statistical historical query records is 24 hours, divided by hour, and the unit time is one hour.

[0054] S3512. Input the heat value results as sample data into the time series prediction model, predict the heat value of each view function in each time interval in the next 24 hours, and establish a set of prediction results.

[0055] S3513. Read all prediction result sets in the current time interval every unit time, sort the query statements in descending order according to the heat value, obtain the view functions with heat values ​​in the top X%, and establish a query task.

[0056] S3514. The query executor executes the view function corresponding to the query task and writes the function execution result into the aggregate result cache collection for query use.

[0057] This embodiment generates a heat value based on historical query records, and then predicts the heat value through a time series prediction model, formulates a query task list based on the prediction results, and then executes the query task. The query task is established by selecting the prediction results with high heat values, so that the execution results are more easily used directly, thereby improving the data query response speed. And through the heat prediction, the task is formulated for scheduled execution, and the cached results are automatically updated without the cost of additional manual maintenance.

[0058] This embodiment also discloses a Mongo query optimization system based on time series prediction and generation, and the system specifically includes a client, a Mongo database, and a database processing unit.

[0059] The database processing unit includes the data process MongObserver, the predicted heat value module and the execution module. The data process MongObserver is used to perform related operations on the Mongo database. The specific database process MongObserver includes the query processor QueryHandler, the query executor QueryExecutor and the query queue QueryQueue.

[0060] The query processor QueryHandler obtains query requests and performs aggregate analysis and judgment. For query requests involving aggregate analysis, hit judgment is made based on cached results, and for cached result misses, the view function is obtained, including direct acquisition if the view function exists, and re-establishment if the view function does not exist. The query results are returned and the query records are written into the query record statistics table.

[0061] Query queue QueryQueue, write query processor QueryHandler and query executor QueryExecutor to obtain view function, and query executor QueryExecutor reads view function.

[0062] The query executor QueryExecutor reads the view function from the query queue QueryQueue for execution.

[0063] The heat value prediction module, at a fixed time every day, based on historical query records, counts the number of requests C for each view function per unit time, the median average of the query response time R, and the number of query users and IPs U, and assigns weights λ, β, and α respectively and then sums them up to obtain the heat value H of each view function in each time interval, H=λC+βR+αU, and construct a heat value set; the heat value results are used as sample data and input into the time series prediction model to predict the heat value of each view function in each time interval in the next 24 hours and establish a prediction result set.

[0064] The execution module reads all the prediction result sets in the current time interval every unit time, sorts the view functions in descending order according to the heat value, obtains the view functions with the heat value in the top X%, establishes a query task, executes the view function corresponding to the query task, and writes the function execution result into the aggregate result cache collection.

[0065] Embodiment 2: This embodiment provides another implementation of a Mongo query optimization method based on time series prediction and generation. Compared with embodiment 1, the query executor QueryExecutor in this embodiment also includes executing the query task immediately, specifically, Figure 4 As shown, the following process is included: S3521. After the query executor QueryExecutor is awakened, the view function in the query queue QueryQueue is executed in a loop until the query queue is empty and the execution result is returned.

[0066] S3522. Return the execution result to the query processor QueryHandler, the processor QueryHandler writes the execution result into the aggregate structure cache collection, inserts the query record into the query record statistics table, and returns the query result to the client.

[0067] In addition, the method of this embodiment specifically includes the following steps: S1. The database process obtains the query request and performs aggregate analysis and judgment; S2. Parse the query request involving aggregate analysis and make a hit judgment based on the cached results; The cached results include the execution results of scheduled query tasks, which are constructed by predicting the popularity of each time period based on historical query records and obtaining the most popular query statements; S3. For a miss, obtain the view function and obtain the query result according to the view function execution result.

[0068] The present invention can generate a heat value according to historical query records, make predictions according to the heat value, formulate query tasks according to the prediction results, execute different query tasks in different time intervals according to the query task list and cache the results, thereby improving the response speed of data query and avoiding manual data processing script operation and maintenance. The query task is generated according to the historical query record, and the larger the historical query record data, the more accurate the prediction. The method of the present invention combines the characteristics of the Mongo database itself, based on the prediction and generation function method, without increasing the cost of software and hardware.

[0069] The Mongo database has a database process MongObserver, which is used to perform related operations on the Mongo database. The database process MongObserver includes a query processor QueryHandler, a query executor QueryExecutor, and a query queue QueryQueue. The above units are applied in subsequent processes involving aggregation analysis and judgment and cache result hit judgment.

[0070] As a preferred solution of this embodiment, step S1 specifically includes the following process: S11. The client Client queries the Mongo proxy service, that is, sends a query request to the database process MongoObserver.

[0071] S12. After the database process MongoObserver receives the query request, its internal query processor QueryHandler evaluates the query request to determine whether the query request involves aggregation analysis.

[0072] S13. If no aggregation analysis is involved, e.g. Figure 1As shown, the query processor QueryHandler forwards the query request to the database, the database performs the query and returns the query result to the query processor QueryHandler, and the query processor QueryHandler returns the query result to the client.

[0073] S14. If aggregation analysis is involved, go to step S2.

[0074] After obtaining the query request, the first step is to determine whether it involves aggregation analysis. If it involves aggregation analysis, it will be processed separately. If it involves aggregation analysis, the corresponding aggregation result cache set will be read to make a cache hit judgment. If it does not involve aggregation analysis, it will be directly sent to the database for query and the query results will be returned, which effectively improves the query speed for complex aggregation analysis in the query.

[0075] As a preferred solution of this embodiment, step S2 specifically includes the following process: S21. The query processor QueryHandler extracts the query statement from the query request, obtains the query target collection name CollName according to the query statement, and calculates the collection status code CollStats, obtains the aggregate result cache collection name CacheColName according to the target collection name CollName, extracts the aggregate statement from the query request, and performs hashing to obtain the query code QueryCode.

[0076] S22. The query processor QueryHandler queries the aggregate cache result in the collection corresponding to the aggregate result cache collection name according to the query code QueryCode and the query target collection status code CollStats, and determines whether there is a cache result with the same query code.

[0077] S23. Figure 2 As shown, if there is a cached result, determine whether the collection status code is the same. If the same, confirm that the cached result is the latest query result, insert the query record into the query statistics table, return the query result to the database process MongoObserver, and the database process MongoObserver returns the query result to the client; if they are not the same, the cached result is not hit, and enter step S3. When each query obtains a query result, the query record is inserted into the query statistics table to form a historical query record, which is used for later heat prediction.

[0078] S24. If there is no cache result, the cache result is not hit, and the process goes to step S3.

[0079] By judging the hit of cached results, the hit cached results are directly returned as query results. By reusing the cached results, there is no need to execute queries from the original database, which effectively improves the data query response speed under large data volumes.

[0080] As a preferred solution of this embodiment, Figure 3 As shown, step S3 specifically includes the following process: S31. The query processor QueryHandler checks whether there is a view function with the same query code and query target collection name in the view function list set QueryViewFuncList by using the query code QueryCode and the query target collection name CollName.

[0081] S32. If the view function exists, write the view function into the query queue QueryQueue and go to step S34.

[0082] S33. If the view function does not exist, create a new view function, and then write the view function into the query queue QueryQueue.

[0083] S34. The query processor QueryHandler wakes up the query executor QueryExecutor to execute the view function in the query queue, wherein the query queue QueryQueue supports data persistence in the database to ensure a fault recovery mechanism.

[0084] S35. After executing the view function, the query executor QueryExecutor returns the execution result to the query processor QueryHandler, which returns the query result to the client Client and inserts the query record of the query request into the query statistics table. When each query obtains the query result, the query record is inserted into the query statistics table to form a historical query record, which is used for later heat prediction.

[0085] As a preferred solution of this embodiment, the query executor QueryExecutor includes a scheduled execution working mode, which executes query tasks regularly and writes them into the aggregated result cache collection. The query task is constructed by predicting the popularity of each time period based on historical query records and obtaining the view function with high popularity. Specifically, Figure 5 As shown, the timing execution includes the following processes: S3511. At a fixed time every day, based on historical query records, count the number of requests C per unit time for each view function, the median average of the query response time R, and the number of query users and IPs U, and assign weights λ, β, and α respectively and sum them up to obtain the heat value H of each view function in each time interval, H=λC+βR+αU, and construct a heat value set. The cycle of statistical historical query records is 24 hours, divided by hour, and the unit time is one hour.

[0086] S3512. Input the heat value results as sample data into the time series prediction model, predict the heat value of each view function in each time interval in the next 24 hours, and establish a set of prediction results.

[0087] S3513. Read all prediction result sets in the current time interval every unit time, sort the query statements in descending order according to the heat value, obtain the view functions with heat values ​​in the top X%, and establish a query task.

[0088] S3514. Execute the view function corresponding to the query task, and write the function execution result into the aggregate result cache collection for query use.

[0089] This embodiment generates a heat value based on historical query records, and then predicts the heat value through a time series prediction model, formulates a query task list based on the prediction results, and then executes the query task. The query task is established by selecting the prediction results with high heat values, so that the execution results are more easily used directly, thereby improving the data query response speed. And through the heat prediction, the task is formulated for scheduled execution, and the cached results are automatically updated without the cost of additional manual maintenance.

[0090] The specific embodiments described herein are merely examples of the spirit of the present invention. Those skilled in the art may make various modifications or additions to the specific embodiments described or replace them in similar ways, but they will not deviate from the spirit of the present invention or exceed the scope defined by the appended claims.

[0091] Although this article uses more terms such as database process, view function, cache result, query task, and predicted result set, it does not exclude the possibility of using other terms. These terms are used only to more conveniently describe and explain the essence of the present invention; interpreting them as any additional restrictions is contrary to the spirit of the present invention.

Claims

1. A Mongo query optimization method based on time series prediction and generation, Features: include, The database process obtains the query request, performs aggregate analysis and judgment, parses the query request involving aggregate analysis, makes a hit judgment based on the cache result, obtains the view function for the miss case, and obtains the query result based on the view function execution result; The cached results include the execution results of scheduled query tasks. The query tasks are constructed by predicting the popularity of each time period based on historical query records and obtaining the most popular view functions.

2. According to claim 1, a Mongo query optimization method based on time series prediction and generation, Its characteristics are: The heat prediction includes: At a fixed time every day, based on historical query records, count the number of requests per view function per unit time, the median average of query response time, the number of query users and IP addresses, assign weights to each, and sum them up to obtain the heat value of each time interval and build a heat value set; The heat value results are used as sample data and input into the time series prediction model to predict the heat value of each view function in each time interval in the next 24 hours and establish a set of prediction results.

3. According to claim 2, a Mongo query optimization method based on time series prediction and generation, Its characteristics are: Get the most popular query statements to build query tasks, including: Read all the prediction result sets in the current time interval, sort the view functions in descending order according to the heat value, obtain the view functions with the heat value in the top X%, and establish a query task.

4. According to claim 3, a Mongo query optimization method based on time series prediction and generation, Its characteristics are: Schedule query tasks, including: Get the query task of the current time interval every unit time, execute the view function corresponding to the query task, and write the function execution result into the aggregate result cache collection.

5. According to claim 4, a Mongo query optimization method based on time series prediction and generation, Its characteristics are: Aggregate analysis and judgment include: Obtain a query request, and evaluate whether aggregation analysis is involved based on the query request; If no aggregate analysis is involved, the query request is forwarded to the database, and the database returns the query result; If aggregation analysis is involved, perform cache result hit judgment.

6. A Mongo query optimization method based on time series prediction and generation according to claim 5, Its characteristics are: Parsing the query request includes: The database process obtains the query request, extracts the query statement, obtains the query target collection name according to the query statement, calculates the collection status code, obtains the aggregate result cache collection name according to the target collection name, extracts the aggregate statement of the query request, and performs hashing to obtain the query code.

7. The Mongo query optimization method based on time series prediction and generation according to claim 6, Its characteristics are: Hit judgment is performed based on the cache structure, including: According to the query code and the query target collection status code, query the aggregate cache result in the collection corresponding to the aggregate result cache collection name to determine whether there is a cache result with the same query code; If there is a cached result, determine whether the collection status code is the same. If so, confirm that the cached result is the latest query result and return the query result. If not, the cached result is not hit and proceed to the step of obtaining the view function. If there is no cached result, the cached result is not hit, and the view function acquisition step is performed.

8. The Mongo query optimization method based on time series prediction and generation according to claim 7, Its characteristics are: Get the view function, including: Use the query code and query target collection name to check whether there is a corresponding view function in the view function list collection. If it exists, write the view function into the query queue. If it does not exist, create a new view function and write the view function into the query queue.

9. A Mongo query optimization method based on time series prediction and generation according to claim 7 or 8, Its characteristics are: After obtaining the query results and completing a query, the query record is inserted into the query record statistics table.

10. A Mongo query optimization method based on time series prediction and generation according to claim 1 or 8, Its characteristics are: The view function execution includes looping and executing the view functions in the query queue until the query queue is empty, executing the view function to obtain the execution result, writing it into the aggregate structure cache collection, and returning the query result.