A log computation derivation time consumption optimization method

Abstract

The present application relates to a kind of log calculation export time-consuming optimization method, and the request that export before needing a lot of time processing integrated data leads to server timeout is divided into multiple requests, first request when in background thread of processing task is established, last request export file, request when in middle state of task is inquired to user show progress. Not only can solve server timeout problem, but also can let user see download progress.

Description

Technical Field

[0001] This invention relates to the field of computers, and more particularly to a method for optimizing the time consumption of log calculation and export. Background Technology

[0002] Because log content is stored in an Elasticsearch database, the log export function requires retrieving the log content and generating a file before exporting the file. The time spent retrieving log content and generating the file can be lengthy, leading to server timeouts. Existing technologies typically address this by setting a server timeout period. However, excessively long timeouts can prevent timely resource release, and even if the time spent retrieving log content and generating the file exceeds the set server timeout, server timeouts will still occur. Summary of the Invention

[0003] In view of this, the present invention provides a method for optimizing log export time to solve or partially solve the above problems.

[0004] To achieve the above-mentioned technical solution, the technical solution of the present invention is as follows: Step 1: Configure the front-end, server, and database, and then set the objective function of the thread scheme to minimize time consumption, with the constraint condition being to avoid server timeout; if the data processing request before exporting logs fails the constraint condition check, the server will decompose the data processing request before exporting logs into multiple requests that can return in a timely manner, and proceed to Step 2; Step 2: When the first request among the multiple requests that can return in a timely manner is executed, a thread for processing tasks will be established on the server; when the last request among the multiple requests that can return in a timely manner is executed, the file will be exported; when the intermediate requests among the multiple requests that can return in a timely manner are executed, the task status will be queried and the progress will be displayed to the user; when the server starts the new thread, it will select... The threading scheme method is as follows: when processing the constraints, server timeout checks need to be performed on all thread nodes. Thread nodes that meet the constraints are added to the root node set, and thread nodes that do not meet the constraints are discarded. The root node set is the set of thread schemes that meet the conditions. The thread nodes are used to run the thread schemes established by the server. The root node is the parent center of the domain search for establishing new thread schemes. The method for optimizing the threads includes: Step T1: The server randomly establishes an initial thread scheme x0 as the primary thread root node. The initial thread scheme must meet the constraints. The objective function value f(x0) of the initial thread scheme x0 is calculated, and the optimal solution of the objective function is x. min =x0, let the initial value of the objective function be f. min (x minStep T2: The server performs a domain search centered on the initial thread scheme x0, extending the primary thread tree to obtain first-level thread nodes; the first-level thread nodes are checked for constraints, and those that do not meet the constraints are directly deleted; Step T3: The server calculates the objective function value of the first-level thread nodes that meet the constraints, and compares it with the initial value f(x0) of the objective function. The first-level thread nodes whose objective function value is greater than the initial value f(x0) of the objective function that meet the constraints are deleted. The server adds the first-level thread nodes that meet the constraints that have not been deleted to the root node set; the server selects the minimum objective function value f of the newly added first-level thread nodes to the root node set. v (x v ) and the f min (x min ) compare, if f v (x v ) is less than the f min (x min If f, then let f min (x min )=f v (x v Let the optimal solution of the objective function be x. min =x v Step T4: The server calculates and saves the optimization rate of the newly added thread nodes in the root node set; Step T5: The server establishes a state probability space in the range [0,1], generates random numbers in the range [0,1], and selects the thread node with the optimization rate equal to the generated random number as the center of the next domain search; Step T6: The server repeats steps T2 to T5 until no unused parent center for the domain search can be selected or the number of iterations specified by the system has been reached; The optimization rate calculation method of the thread nodes in step T4 is as follows: The server establishes a primary thread root node S0, and the primary thread root node grows a primary thread trunk M. It is assumed that there are k first-level thread nodes on the primary thread trunk M. The optimization rate calculation formula for the first-level thread node is as follows: Formula 1; where i is the thread node number on the primary thread tree M, and its value is a positive integer from 1 to k; k is the number of thread nodes on the primary thread tree M, and its value is a positive integer; S is the number of thread nodes, is the thread node numbered i on the primary thread trunk M; f is the objective function, which is the design value; f(S0) is the objective function value of the primary thread root node. P is the objective function value of the thread node numbered i on the primary thread trunk M; P is the optimization rate. It is the optimization rate of the thread node numbered i on the primary thread trunk M; because Optimization rate of all first-level thread nodes The state probability space is composed of [0,1]; the server randomly generates random numbers in the range [0,1], and the random numbers fall within the range of [0,1]. Within a certain state probability space, the first-level thread node that falls into the state probability space. The first-level thread root node S1 is selected first, and a first-level trunk thread R grows from the first-level thread root node. It is assumed that there are q second-level thread nodes on the first-level trunk thread R. The environment of the primary thread node has also changed, and the optimization rate needs to be reallocated, such as: Where j is the number of the thread node on the first-level thread tree R, and its value is a positive integer from 1 to q; q is the number of thread nodes on the first-level thread tree R, and its value is a positive integer. It is the thread node numbered j on the first-level thread tree trunk R; It is the thread node numbered j on the first-level thread tree trunk R; It is the optimization rate of the thread node numbered j on the first-level thread trunk R; The root node of the newly grown thread tree will be removed from the root node set, and the newly grown thread node will be added to the root node set. The growth process is repeated until there are no new thread trees. The front end is connected to the server. The server is connected to the database. The user initiates an export task through the front end, which is also used to display the export task progress and system failures to the user. Step 3, the main steps of log export are as follows: Step S1: After the user clicks the export button provided by the front end, the front end interface sends a request for pre-export log processing data to the server. Step S2: After receiving the request from the front end, the server generates an export identifier ID and corresponding status. The server stores the export identifier ID and... The corresponding status is stored in the cache and returned. The export identifier ID corresponds to the processing status of the new thread. At the same time, the server starts a new thread to query the database and integrate the data. The new thread continuously updates the status corresponding to the export identifier ID during the query process. Step S3: After receiving the export identifier ID, the front end queries the status corresponding to the export identifier ID every 2 seconds and displays the export progress to the user until the status is returned as complete. Step S4: After the thread finishes processing, the server sends a processing completion status to the front end. After receiving the processing completion status, the front end calls the export file request. Step S5: The server streams the file to the front end, prompts the user to save, and deletes the file from the server. Detailed Implementation

[0005] To make the technical problem to be solved, the technical solution, and the beneficial effects of this invention clearer, the invention will be described in detail below with reference to embodiments. It should be noted that the specific embodiments described herein are only for explaining the invention and are not intended to limit the invention. Products that can achieve the same function are equivalent substitutions and improvements, and are all included within the protection scope of this invention. The specific method is as follows:

[0006] Example 1: This example specifically introduces commonly used methods for optimizing log export time, as follows:

[0007] Methods to optimize log export time include: front-end, server, and database;

[0008] The front end connects to the server; the server connects to the database; users initiate export tasks through the front end, which is also used to display the export task progress and system failures to users.

[0009] The main steps for exporting logs are as follows:

[0010] Step S1: After the user clicks the export button provided by the front end, the front end sends a request to the server for processing data before exporting the logs;

[0011] Step S2: After receiving the request from the front end, the server generates an identifier ID and its corresponding status. The server stores the identifier ID and its corresponding status in the cache and returns them. At the same time, the server starts a new thread to query the database and integrate the data. During the query process, the new thread continuously updates the status corresponding to the identifier ID.

[0012] Step S3: After receiving the identifier ID, the front end queries the status corresponding to the identifier ID every 2 seconds and displays the export progress to the user;

[0013] Step S4: After the server finishes processing the thread, it sends a processing completion status to the front end; after receiving the processing completion status, the front end calls the export file request;

[0014] Step S5: The server streams the file to the front end, prompts the user to save it, and deletes the file from the server.

[0015] The objective of the threading scheme is to minimize time consumption, and the constraint is to avoid server timeout. If the data processing request before exporting logs fails the constraint check, the server will decompose the data processing request before exporting logs into multiple requests that can return in a timely manner. The first request creates a thread for processing tasks on the server, the last request exports the file, and the intermediate requests query the status of the tasks to show the progress to the user.

[0016] The method for selecting a thread scheme when the server starts a new thread is as follows: When handling constraints, all thread nodes need to undergo server timeout checks. Thread nodes that meet the constraints are added to the root node set, while thread nodes that do not meet the constraints are discarded. The root node set is the set of thread schemes that meet the conditions. A thread node is a thread scheme established by the server. The root node is the center for the domain search when establishing a new thread scheme. The method for optimizing threads includes the following steps:

[0017] Step T1: The server randomly creates an initial thread scheme x0 as the primary thread root node. The initial thread scheme must satisfy the constraints. Calculate the objective function value f(x0) of the initial thread scheme x0, and let the optimal solution of the objective function be x. min =x0, let the initial value of the objective function be f. min (x min ) = f(x0);

[0018] Step T2: The server performs a domain search centered on the initial thread scheme x0, extending the primary thread tree to obtain first-level thread nodes; the constraints of the first-level thread nodes are checked, and first-level thread nodes that do not meet the constraints are directly deleted.

[0019] Step T3: The server calculates the objective function value of the first-level thread nodes that satisfy the constraints and compares it with the initial value f(x0) of the objective function. First-level thread nodes whose objective function value is greater than the initial value f(x0) that satisfy the constraints are deleted. The server adds the remaining first-level thread nodes that satisfy the constraints to the root node set. The server then selects the minimum objective function value f of the newly added first-level thread nodes to the root node set. v (x v ) and f min (x min ) compare, if f v (x v (less than f) min (x min If f, then let f min (x min )=f v (x v Let the optimal solution of the objective function be x. min =x v ;

[0020] Step T4: The server calculates and saves the optimization rate of the newly added thread nodes in the root node set;

[0021] Step T5: The server establishes a state probability space in the range [0,1] and generates random numbers in the range [0,1]. The thread node corresponding to the optimization rate that is equal to the generated random number is selected as the center of the next domain search.

[0022] Step T6: The server repeats steps T2-T5 until no unused center of the domain search can be selected or the number of iterations specified by the system has been reached;

[0023] The optimization rate calculation method for thread nodes in step T4 is as follows: The server establishes a primary thread root node S0, and the primary thread root node grows a primary thread trunk M. It is assumed that there are k first-level thread nodes on the primary thread trunk M. The optimization rate calculation formula for first-level thread nodes is as follows:

[0024] Formula 1:

[0025] Where i is the index of the thread node on the primary thread tree M, and its value is a positive integer from 1 to k; k is the number of thread nodes on the primary thread tree M, and its value is a positive integer; S is the number of thread nodes, is the thread node with index i on the primary thread trunk M; f is the objective function, which is the design value; f(S0) is the objective function value of the primary thread root node. is the objective function value of thread node i on the primary thread tree M; P is the optimization rate. It is the optimization rate of the thread node numbered i on the primary thread trunk M;

[0026] because Optimization rate of all first-level thread nodes The state probability space is composed of [0,1]; the server randomly generates random numbers in the range [0,1], and the random numbers fall within this range. Within a certain state probability space, the first-level thread node that falls into the state probability space. The first-level thread S1 is selected to become the first-level root node. First-level trunk threads R grow from the first-level root node. Assume that there are q second-level thread nodes on the first-level trunk thread R. The environment of the first-level thread node has also changed, and the optimization rate needs to be reallocated, as follows:

[0027]

[0028] Where j is the number of the thread node on the first-level thread tree trunk R, and its value is a positive integer from 1 to q; q is the number of thread nodes on the first-level thread tree trunk R, and its value is a positive integer. It is the thread node numbered j on the first-level thread tree trunk R; It is the thread node numbered j on the first-level thread tree trunk R; It is the optimization rate of the thread node numbered j on the first-level thread tree trunk R;

[0029] The root node of a new thread tree trunk will be removed from the root node set, while the newly grown thread node will be added to the root node set. The growth process is repeated until there are no new thread tree trunks.

[0030] The beneficial results of this invention are as follows: This invention provides a method for optimizing the time consumption of log calculation and export, including a front-end, a server, and a database. For requests that require a large amount of time to process and integrate data before export, causing server timeouts, the method divides them into multiple requests. During the first request, a thread is created in the background to process the task, and the last request exports the file. During intermediate requests, the task status is queried to display the progress to the user. This not only solves the server timeout problem but also allows users to see the download progress.

[0031] The above are merely preferred embodiments of the present invention and are not intended to limit the scope of protection of the claims. Furthermore, the above description should be understood and implemented by those skilled in the art; therefore, any equivalent modifications made based on the disclosure of this invention should be included within the scope of these claims.

Claims

1. A method for optimizing the time consumption of log calculation and export, characterized in that, This includes: Step 1, configuring the front-end, server, and database, and then setting the objective function of the thread scheme to minimize time consumption, with the constraint being to avoid server timeout; if the data processing request before exporting logs cannot pass the constraint check, the server will decompose the data processing request before exporting logs into multiple requests that can return in a timely manner, and proceed to Step 2. Step 2: For the first request among multiple requests that return in a timely manner, a new thread for processing the task is created on the server. For the last request among multiple requests that return in a timely manner, the file is exported. For intermediate requests among multiple requests that return in a timely manner, the task status is queried and the progress is displayed to the user. The method for selecting the thread scheme when the server starts the new thread is as follows: When processing the constraints, all thread nodes need to be checked for server timeout. Thread nodes that meet the constraints are added to the root node set, and thread nodes that do not meet the constraints are discarded. The root node set is a set of thread schemes that meet the conditions; the thread node is used to run the thread scheme established by the server; the root node is the parent center for neighborhood search of establishing a new thread scheme; the method for optimizing the thread includes: Step T1: The server randomly establishes an initial thread scheme. As the primary thread root node, the initial thread scheme must satisfy the aforementioned constraints; the initial thread scheme is then determined. The objective function value f( Let the optimal solution of the objective function be... Let the initial value of the objective function be... ; Step T2: The server uses the initial thread scheme. A neighborhood search is performed around the center to extend the primary thread tree, resulting in first-level thread nodes. Constraints are checked on these first-level thread nodes; those that do not meet the constraints are directly deleted. Step T3: The server calculates the objective function value of the first-level thread nodes that satisfy the constraints and compares it with the initial value f(of the objective function). The values ​​of the objective functions are compared, and the objective function value is greater than the initial value f( of the objective function). If a first-level thread node that satisfies the constraints is deleted, the server will add the first-level thread nodes that satisfy the constraints that were not deleted to the root node set. The server selects the minimum objective function value of the first-level thread nodes newly added to the root node set. and the If a comparison is made, Smaller than the Then let Let the optimal solution of the objective function be ; Step T4: The server calculates and saves the optimization rate of the newly added thread nodes in the root node set; Step T5: The server establishes a state probability space in the range [0, 1], generates random numbers in the range [0, 1], and selects the thread node with the optimization rate equal to the generated random number as the center of the next neighborhood search; Step T6: The server repeats steps T2 to T5 until no unused parent center for the neighborhood search can be selected or the number of iterations specified by the system has been reached; The optimization rate calculation method of the thread nodes in step T4 is as follows: The server establishes a primary thread root node S0, and the primary thread root node grows a primary thread trunk M. It is assumed that there are k first-level thread nodes on the primary thread trunk M. The optimization rate calculation formula for the first-level thread node is as follows: （1） Where i is the thread node number on the primary thread tree M, and its value is a positive integer from 1 to k; k is the number of thread nodes on the primary thread tree M, and its value is a positive integer; S is the number of thread nodes. is the thread node numbered i on the primary thread trunk M; f is the objective function, which is the design value; f(S0) is the objective function value of the primary thread root node. is the objective function value of the thread node numbered i on the primary thread trunk M; P is the optimization rate. It is the optimization rate of the thread node numbered i on the primary thread trunk M; because Optimization rate of all first-level thread nodes , ,..., The state probability space is composed of [0,1]; the server randomly generates random numbers in the range [0,1], and the random numbers fall within the range of [0,1]. , ,..., Within a certain state probability space, the first-level thread node that falls into the state probability space. The first-level thread root node S1 is selected first, and a first-level thread tree trunk R grows from the first-level thread root node. It is assumed that there are q second-level thread nodes on the first-level thread tree trunk R. , ,..., The environment of the first-level thread node also changes, and the optimization rate needs to be reallocated using the new formula: （2） Where j is the number of the thread node on the first-level thread tree R, and its value is a positive integer from 1 to q; q is the number of thread nodes on the first-level thread tree R, and its value is a positive integer. It is the thread node numbered j on the first-level thread tree trunk R; It is the thread node numbered j on the first-level thread tree trunk R; It is the optimization rate of the thread node numbered j on the first-level thread trunk R; The root node of the newly grown thread tree will be removed from the root node set, and the newly grown thread node will be added to the root node set. The growth process is repeated until there are no new thread trees. The front end is connected to the server. The server is connected to the database. The user initiates an export task through the front end, which is also used to display the export task progress and system failures to the user. Step 3, the log export steps are as follows: Step S1: After the user clicks the export button provided by the front end, the front end interface sends a processing data request to the server before exporting the logs. Step S2: After receiving the request from the front end, the server generates an export identifier ID and a corresponding status. The server stores the export identifier ID and... The corresponding status is stored in the cache and returned. The export identifier ID corresponds to the processing status of the new thread. At the same time, the server starts a new thread to query the database and integrate the data. The new thread continuously updates the status corresponding to the export identifier ID during the query process. Step S3: After receiving the export identifier ID, the front end queries the status corresponding to the export identifier ID every 2 seconds and displays the export progress to the user until the status is returned as complete. Step S4: After the thread finishes processing, the server sends a processing completion status to the front end. After receiving the processing completion status, the front end calls the export file request. Step S5: The server streams the file to the front end, prompts the user to save, and deletes the file from the server.

Images