A memory overflow early warning method and device, electronic equipment, and storage medium
By classifying and calculating the set of query statements and combining historical data to optimize memory overflow warnings, the problems of high false alarm rate and resource waste have been solved, and more accurate and efficient memory overflow warnings have been achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- CHINA CONSTRUCTION BANK
- Filing Date
- 2022-09-07
- Publication Date
- 2026-04-24
AI Technical Summary
Existing memory overflow warning methods have a high false alarm rate and cause serious waste of system resources, mainly because they do not take into account the differences in executed statements and the regularity of calculations.
By obtaining a set of query statements within the target time period, classifying and calculating the expected memory usage and peak execution frequency of each type of statement, combining historical data for weighted summation, calculating the total peak memory usage, and providing early warning information.
It reduced the false alarm rate of memory overflow, optimized system resource utilization, and improved the accuracy and efficiency of memory overflow warnings.
Smart Images
Figure CN115629946B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of data processing technology, and in particular to a method and apparatus for early warning of memory overflow, electronic equipment, and storage medium. Background Technology
[0002] With the development of Internet technology, server programs need to execute statements to retrieve data from the database, then generate objects from the data for processing. These objects also occupy a portion of memory space. If the objects occupy too much memory, it can easily cause memory shortage problems, leading to system crashes. Therefore, it is necessary to provide timely warnings about memory overflow.
[0003] In existing technologies, two main methods are used for memory overflow warnings. One method is static code analysis, which checks whether the executable statements in the source code match corresponding rules or regular expression patterns. If they match, it determines whether any executable statement controls the number of records in the returned dataset. If it is found that the executable statement does not control the number of records in the returned dataset, it indicates that the data in the relevant data list may be excessive, and this situation needs to be reported to relevant personnel for warning processing. The other method combines memory monitoring and object monitoring technologies, which simultaneously monitor memory usage and target objects to determine whether the target object's expected memory usage exceeds a threshold. If it does, a warning is issued.
[0004] However, the first method, due to its static code analysis technology's lack of consideration for differences in executed statements and its weak targeting, suffers from a high false alarm rate for memory overflow. The second method, with its memory monitoring and object monitoring technologies, only considers the current memory status and the memory usage of the target object, failing to systematically account for computational patterns. This leads to excessive system resource consumption and low system efficiency. Summary of the Invention
[0005] In view of the shortcomings of the prior art, this application provides a method and device for early warning of memory overflow, an electronic device, and a storage medium to solve the problem of high false alarm rate of memory overflow in the prior art.
[0006] To achieve the above objectives, this application provides the following technical solution:
[0007] The first aspect of this application provides a method for early warning of memory overflow, including:
[0008] Retrieve the collection of current query statements of various types within the target time period;
[0009] Calculate the expected memory usage of the current query statement set for each of the aforementioned types to obtain the current expected value for each type.
[0010] Based on the current expected value corresponding to each of the aforementioned types, determine the current target expected value corresponding to each of the aforementioned types;
[0011] Calculate the peak execution frequency of the current query statement set for each type to obtain the peak execution frequency for each type.
[0012] Based on the current execution frequency peak value corresponding to each of the aforementioned types, determine the current target execution frequency peak value corresponding to each of the aforementioned types;
[0013] Based on the current target expected value and the corresponding current target execution frequency peak value of each type, calculate the peak memory usage of each type;
[0014] Based on the peak memory usage corresponding to each type and its corresponding current target expected value, calculate the total peak memory usage within the target time period;
[0015] If the total peak value is not less than the first preset threshold, then a warning message is sent to the front end.
[0016] Optionally, in the above-described memory overflow warning method, obtaining the set of current query statements of various types within the target time period includes:
[0017] Retrieve all current query statements within the target time period;
[0018] According to preset rules, each current query statement is classified to obtain a set of current query statements of each type;
[0019] For each of the aforementioned types, the set of current query statements for that type is identified using the common feature values of the current query statements for that type.
[0020] Optionally, in the above-described memory overflow warning method, determining the current target expected value for each type based on the current expected value for each type includes:
[0021] Based on the common feature values of the current query statement sets of each type, check whether there are historical target expected values corresponding to each type;
[0022] If no historical target expected value is found for each of the aforementioned types, then the current expected value for each of the aforementioned types is determined as the current target expected value for each of the aforementioned types.
[0023] If historical target expected values are found for each of the aforementioned types, then the current expected value for each type and its corresponding historical target expected value are weighted and summed to obtain the current target expected value for each type.
[0024] Optionally, in the above-described memory overflow warning method, determining the current target execution frequency peak value corresponding to each of the aforementioned types based on the current execution frequency peak value includes:
[0025] Based on the common feature values of the current query statement sets of each type, search for whether there are historical target execution frequency peaks corresponding to each type;
[0026] If no historical execution frequency peak is found for each of the aforementioned types, then the current execution frequency peak for each of the aforementioned types is determined as the current target execution frequency peak for each of the aforementioned types.
[0027] If historical target execution frequency peaks corresponding to each of the aforementioned types are found, then the current target execution frequency peaks corresponding to each of the aforementioned types and their corresponding historical target execution frequency peaks are weighted and summed to obtain the current target execution frequency peaks corresponding to each of the aforementioned types.
[0028] Optionally, in the above-described memory overflow warning method, the step of calculating the expected memory usage of the current query statement set for each type to obtain the current expected value corresponding to each type includes:
[0029] The memory usage of each current query statement of each type is obtained by aspect-based approach;
[0030] Calculate the average memory usage of each current query statement for each type to obtain the average value corresponding to each type;
[0031] Obtain the total number of samples of the current query statement set of each type within the target time period and the number of time units within the target time period;
[0032] Based on the total number of samples and the number of time units within the target time period, the execution frequency of the current query statement set of each type is calculated to obtain the execution frequency corresponding to each type.
[0033] Based on the average value and execution frequency of each type, the expected memory usage of the current query statement set for each type is calculated using the target algorithm, thus obtaining the current expected value for each type.
[0034] Optionally, in the above-described memory overflow warning method, the step of calculating the peak execution frequency of the current query statement set for each type to obtain the peak execution frequency corresponding to each type includes:
[0035] Obtain the execution timestamp of each of the current query statements of each of the aforementioned types;
[0036] Based on the execution timestamp of each current query statement of each type, calculate the maximum number of times each current query statement of each type is executed in each time unit of the target time period, and obtain the maximum number of times corresponding to each type;
[0037] The maximum number of times corresponding to each type is determined as the current execution frequency peak value corresponding to each type.
[0038] Optionally, in the above-described memory overflow warning method, before calculating the peak execution frequency of the current query statement set for each type to obtain the peak execution frequency corresponding to each type, the method further includes:
[0039] Determine whether there is a current target expected value that is less than a second preset threshold among the current target expected values corresponding to each type;
[0040] If it is determined that there is a current target expected value less than the second preset threshold among the current target expected values corresponding to each of the types, then the set of current query statements corresponding to the types with current target expected values less than the second preset threshold will be deleted.
[0041] A second aspect of this application provides a memory overflow early warning device, comprising:
[0042] The first acquisition unit is used to acquire a set of current query statements of various types within the target time period;
[0043] The first calculation unit is used to calculate the expected value of memory occupied by the current query statement set of each type, and obtain the current expected value corresponding to each type;
[0044] The first determining unit is used to determine the current target expected value corresponding to each of the types based on the current expected value corresponding to each type;
[0045] The second calculation unit is used to calculate the peak execution frequency of the current query statement set for each type, and obtain the peak execution frequency corresponding to each type.
[0046] The second determining unit is used to determine the current target execution frequency peak value corresponding to each of the types based on the current execution frequency peak value corresponding to each type.
[0047] The third calculation unit is used to calculate the peak memory usage corresponding to each type based on the current target expected value and the corresponding current target execution frequency peak value of each type.
[0048] The fourth calculation unit is used to calculate the total peak memory usage within the target time period based on the peak memory usage corresponding to each type and its corresponding current target expected value.
[0049] The feedback unit is used to send a warning message to the front end if the total peak value is not less than a first preset threshold.
[0050] Optionally, in the above-described memory overflow warning device, the first acquisition unit includes:
[0051] The second acquisition unit is used to acquire each current query statement within the target time period;
[0052] The classification unit is used to classify each current query statement according to preset rules to obtain a set of current query statements of each type;
[0053] The identification unit is used to identify the set of current query statements of each type by using the common feature value of each current query statement of the type.
[0054] Optionally, in the above-described memory overflow warning device, the first determining unit includes:
[0055] The first search unit is used to search for whether there is a historical target expected value corresponding to each type based on the common feature value of the current query statement set of each type.
[0056] The third determining unit is used to determine the current expected value corresponding to each type as the current expected value corresponding to each type if no historical target expected value is found for each type.
[0057] The first weighted summation unit is used to, if it finds that there are historical target expected values corresponding to each of the types, perform a weighted summation of the current expected value corresponding to each type and its corresponding historical target expected value to obtain the current target expected value corresponding to each type.
[0058] Optionally, in the above-described memory overflow warning device, the second determining unit includes:
[0059] The second search unit is used to search for the existence of historical target execution frequency peaks corresponding to each type based on the common feature values of the current query statement sets of each type.
[0060] The fourth determining unit is used to determine the current execution frequency peak corresponding to each type as the current target execution frequency peak corresponding to each type if no historical execution frequency peak corresponding to each type is found.
[0061] The second weighted summation unit is used to, if it is found that there are historical target execution frequency peaks corresponding to each type, perform weighted summation on the current execution frequency peaks corresponding to each type and their corresponding historical target execution frequency peaks to obtain the current target execution frequency peaks corresponding to each type.
[0062] Optionally, in the above-described memory overflow warning device, the first computing unit includes:
[0063] The third acquisition unit is used to acquire the memory usage value of each current query statement of each type through aspect-based methods;
[0064] The fifth calculation unit is used to calculate the average memory usage of each current query statement of each type, so as to obtain the average value corresponding to each type;
[0065] The fourth acquisition unit is used to acquire the total number of samples of each type of current query statement set within the target time period and the number of time units within the target time period;
[0066] The sixth calculation unit is used to calculate the execution frequency of the current query statement set of each type based on the total number of samples and the number of time units within the target time period, so as to obtain the execution frequency corresponding to each type.
[0067] The seventh calculation unit is used to calculate the expected value of the memory occupied by the current query statement set of each type based on the average value and the corresponding execution frequency of each type, and to obtain the current expected value corresponding to each type.
[0068] Optionally, in the above-described memory overflow warning device, the second computing unit includes:
[0069] The fifth acquisition unit is used to acquire the execution timestamp of each current query statement of each type;
[0070] The eighth calculation unit is used to calculate the maximum number of times each of the current query statements of each type is executed within each time unit of the target time period based on the execution timestamp of each of the current query statements of each type, so as to obtain the maximum number of times corresponding to each type.
[0071] The fifth determining unit is used to determine the maximum number of times corresponding to each type as the current execution frequency peak value corresponding to each type.
[0072] Optionally, the aforementioned memory overflow warning device further includes:
[0073] The judgment unit determines whether there is a current target expected value that is less than a second preset threshold among the current target expected values corresponding to each type;
[0074] The deletion unit is used to delete the set of current query statements corresponding to the types of current target expected values that are less than the second preset threshold if it is determined that there is a current target expected value less than the second preset threshold among the current target expected values corresponding to each type.
[0075] A third aspect of this application provides an electronic device, comprising:
[0076] Memory and processor;
[0077] The memory is used to store programs;
[0078] The processor is used to execute the program, which, when executed, is specifically used to implement a memory overflow warning method as described in any of the above.
[0079] The fourth aspect of this application provides a computer storage medium for storing a computer program, which, when executed, implements a memory overflow warning method as described in any of the preceding claims.
[0080] This application provides a memory overflow early warning method. It obtains a set of current query statements of various types within a target time period, calculates the expected memory usage for each type of current query statement set, and then determines the current target expected value for each type. Next, it calculates the peak execution frequency of each type of current query statement set, and then determines the current target execution frequency peak for each type. Finally, it calculates the peak memory usage for each type based on the current target expected value and its corresponding peak execution frequency, and then calculates the total peak memory usage within the target time period based on the peak memory usage and the corresponding current target expected value. If the total peak value exceeds a first preset threshold, an early warning message is sent to the front end. This method solves the problem of a high false alarm rate for memory overflows due to neglecting the differences in executed statements, and also addresses the problem of excessive system resource consumption due to not systematically considering the regularity of calculations. Attached Figure Description
[0081] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.
[0082] Figure 1 A flowchart illustrating a memory overflow early warning method provided in this application embodiment;
[0083] Figure 2 A flowchart illustrating a method for obtaining a set of query statements provided in an embodiment of this application;
[0084] Figure 3 A flowchart illustrating a method for obtaining a current expected value, provided in an embodiment of this application;
[0085] Figure 4 A flowchart illustrating a method for obtaining a current target expected value, provided in an embodiment of this application;
[0086] Figure 5 A flowchart illustrating a method for obtaining the current execution frequency peak value provided in an embodiment of this application;
[0087] Figure 6 A flowchart illustrating a method for obtaining the peak execution frequency of a current target, as provided in an embodiment of this application;
[0088] Figure 7 A schematic diagram of a memory overflow early warning device provided in another embodiment of this application;
[0089] Figure 8 This is a schematic diagram of the structure of an electronic device provided in another embodiment of this application. Detailed Implementation
[0090] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0091] In this application, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.
[0092] This application provides a method for early warning of memory overflow, such as... Figure 1 As shown, the specific steps include:
[0093] S101. Obtain the collection of current query statements of various types within the target time period.
[0094] It should be noted that the target time period refers to the time period in which the system is used most frequently. The system obtains the set of current query statements formed by the various query statements within the target time period.
[0095] Optionally, in another embodiment of this application, one specific implementation of step S101 is as follows: Figure 2 As shown, it includes the following steps:
[0096] S201. Obtain all current query statements within the target time period.
[0097] Specifically, the system can obtain various query statements within the target time period through various methods such as querying logs or setting up data collection devices.
[0098] S202. According to preset rules, classify each current query statement to obtain a set of current query statements of each type.
[0099] Optionally, the preset rule can be that when any two query statements have the same table and syntax features, but the values corresponding to the query conditions are different, such as: select * from user where name = jimmy and select * from user where name = lucy, these two query statements are only different in terms of the values corresponding to the query conditions, so they can be regarded as the same type of statement and placed in the same set.
[0100] Alternatively, since the query statements in the current application system's database processing framework (such as commonly used ORM frameworks like Hibernate and Mybatis) all have fixed ID identifiers, query statements with a common ID identifier can also be grouped into the same set.
[0101] S203. For each type, identify the set of current query statements for that type by using the common feature values of each current query statement for that type.
[0102] It should be noted that step S203 is performed in order to find the historical calculation results corresponding to each type based on the common feature values of each type. Therefore, it is necessary to use the common feature values of each current query statement of each type to identify the current query statement set of each type.
[0103] Alternatively, the common feature values of each current query statement can be extracted by string extraction or by using a tool to parse and process each current query statement to obtain common feature values. For example, Java provides the jsqlparser tool, which can extract key information (table name, WHERE condition, etc.) of each current query statement and then use the extracted key information as the feature values of each current common query statement.
[0104] S102. Calculate the expected memory usage of the current query statement set for each type, and obtain the current expected value for each type.
[0105] The expected memory usage refers to the average memory space occupied by the set of query objects corresponding to the set of query statements. Specifically, the most direct way to obtain the memory usage of the set of query objects is to execute the query statement corresponding to that query object. The memory usage of the query object can only be calculated when the query object is in memory. Therefore, for any set of query statements, it is necessary to first obtain the memory usage of the query objects assigned after the execution of each query statement in the set of query statements, and then calculate the expected value of the set of query statements based on each memory value.
[0106] Optionally, in another embodiment of this application, one specific implementation of step S102 is as follows: Figure 3 As shown, it includes the following steps:
[0107] S301. Obtain the memory usage value of each current query statement of each type through aspect-based methods.
[0108] Specifically, aspect-oriented programming can be used to insert code that calculates the memory usage of the query object at the point where the query statement is executed. After the calculation, the memory usage of the query statement on the corresponding query object can be recorded through logging or other methods. It should be noted that among the types obtained by aspect-oriented programming, primitive types and unstructured types can be removed, and only structured type objects such as List, Set, and Map can be obtained. This is because the memory usage of a single object and primitive types is generally negligible, and only the multiple rows of data returned by the corresponding query statement need to be considered.
[0109] S302. Calculate the average memory usage of each current query statement for each type to obtain the average value for each type.
[0110] S303. Obtain the total number of samples of each type of current query statement set within the target time period and the number of time units within the target time period.
[0111] It should be noted that in this embodiment, the memory usage value is calculated periodically, which is why it is necessary to obtain the total number of samples and the number of time units within the target time period in order to accurately calculate the memory usage value.
[0112] S304. Based on the total number of samples and the number of time units within the target time period, calculate the execution frequency of the current query statement set for each type, and obtain the execution frequency corresponding to each type.
[0113] Specifically, the calculation expressions for the execution frequency corresponding to each type are as follows:
[0114] Total number of samples within the target time period ÷ Number of time units within the target time period
[0115] For example: if the target time period is 1 hour and the time unit is 1 second, then the number of time units within the target time period is 3600. Therefore, the number of samples of a certain current query statement set within this 1 hour is 7200, and the execution frequency of the current query statement is 7200÷3600=2.
[0116] Specifically, if the calculated execution frequency is less than 1, then the execution frequency is set to 1.
[0117] S305. Based on the average value and execution frequency of each type, use the target algorithm to calculate the expected memory occupied by the current query statement set of each type, and obtain the current expected value of each type.
[0118] Specifically, the calculation method for the current expected value corresponding to each type is as follows:
[0119] Expected memory usage = Average memory usage × Execution frequency
[0120] S103. Based on the current expected value corresponding to each type, determine the current target expected value corresponding to each type.
[0121] Optionally, in another embodiment of this application, one specific implementation of step S103 is as follows: Figure 4 As shown, it includes the following steps:
[0122] S401. Based on the common feature values of the current query statement sets of each type, check whether there are corresponding historical target expected values for each type.
[0123] It should be noted that the early warning mechanism in this embodiment requires periodic sampling and calculation. Subsequent calculations may need to reference and integrate the results of the previous calculation. Each round of calculation retains useful information for subsequent calculations. This useful information may include the expected memory usage, peak values, common characteristic values, etc., of each type, thereby improving the effectiveness of the early warning mechanism and making memory overflow warnings more accurate. Of course, not every type will have its previous calculation process for subsequent calculations, and it may be the first time it is being calculated. Therefore, it is necessary to search for the existence of historical target expected values for each type based on the common characteristic values of the current query statement set for each type. If no historical target expected values are found for each type, it indicates that each type may be being calculated for the first time, or the previous calculation results may not be effective enough. Therefore, step S402 needs to be executed. If historical target expected values are found for each type, it indicates that the current expected values for each type need to be integrated and referenced with the previous calculation results. Therefore, step S403 needs to be executed.
[0124] S402. Determine the current expected value corresponding to each type as the current target expected value corresponding to each type.
[0125] S403. The current expected value and its corresponding historical target expected value for each type are weighted and summed to obtain the current target expected value for each type.
[0126] Specifically, the weighted summation of the current expected value for each type calculated in S102 and the corresponding historical expected value retained yields the following expression for calculating the integrated current expected value for each type:
[0127] X = x1*f1 + x2*f2 (when x1 > x2)
[0128] X = x1*f2 + x2*f1 (when x1 <x2)
[0129] In particular, when x1 = 0, X = x2.
[0130] Where x1 is the historical target expected value corresponding to each type (0 if it does not exist), x2 is the current expected value corresponding to each type (0 if it does not exist), X is the current target expected value corresponding to each type, f1 and f2 are weights, and f1>f2, f1+f2=1. The weights can be set according to the requirements, but the conditions must be met, and this calculation method tends to retain the larger expected value obtained from the two sampling calculations.
[0131] Then, when x2 = 0, X = x1 * f1, which means that no corresponding query statement was found in the current query statement set of each type. In this case, the current target expectation value X needs to be assigned as the historical target expectation value corresponding to each type multiplied by a weight less than 1. This is because this method can continuously lower the expectation value of query statements that have not been executed for a long time, and eventually exclude them from the calculation range, making the early warning mechanism flexible and changeable.
[0132] It should be noted that the current target expectation value also needs to be used in subsequent calculations and retained in the historical sets of various types as the historical target expectation value used in the next round of calculations.
[0133] S104. Calculate the peak execution frequency of the current query statement set for each type to obtain the peak execution frequency for each type.
[0134] It should be noted that in this embodiment of the application, the calculation process takes into account the multi-dimensional influencing factors (SQL execution frequency, SQL execution frequency peak, etc.) which may lead to the final calculation result being inaccurate. Therefore, in order to optimize the calculation process in a targeted manner and make the subsequent calculation result sufficiently accurate, the execution frequency peak of the query statement set is also taken into account in the calculation, so that the peak memory usage can be calculated flexibly and efficiently.
[0135] Optionally, in another embodiment of this application, one specific implementation of step S104 is as follows: Figure 5 As shown, it includes the following steps:
[0136] S501. Obtain the execution timestamp of each current query statement of each type.
[0137] It should be noted that the execution timestamp refers to the execution time of the query statement. Based on the execution timestamp, we can know whether the query statement was executed within a certain time period, and thus know how many types of queries were executed within that time period.
[0138] S502. Based on the execution timestamp of each current query statement of each type, calculate the maximum number of times each current query statement of each type is executed in each time unit of the target time period, and obtain the maximum number of times corresponding to each type.
[0139] Specifically, if any type of current query statement is executed 100 times within the time period from 12:37:33 to 12:37:34, which is the maximum number of times it is executed within this second, then the current frequency peak of this type is considered to be 100.
[0140] S503. Determine the maximum number of times corresponding to each type as the current execution frequency peak value corresponding to each type.
[0141] Optionally, in the application embodiment, after performing step S103 and before step S104, the process further includes:
[0142] Determine whether there is a current target expected value that is less than the second preset threshold among the current target expected values corresponding to each type.
[0143] It should be noted that, in order to accurately calculate memory usage, this embodiment specifically considers the following: if the expected value (memory usage) is small and the execution frequency is high, the memory objects generated by the query will exist in memory for a short time and will be quickly destroyed after use. Therefore, it can be assumed that there will not be a large number of such objects at the same time, and the high execution frequency generally will not cause excessive memory consumption. If a query consumes a lot of memory, although its execution frequency is low, its execution at a certain moment will instantly consume a lot of memory, and the execution time will generally be long. Therefore, it is preferable to retain such data for subsequent calculations. However, in order to simplify the computational workload and improve the computational efficiency of the system, some query statement sets with relatively small memory consumption can be filtered out. Therefore, a filter value will be set accordingly to determine the current target expected value corresponding to each type, that is, to determine whether there is a current target expected value less than a second preset threshold among the current target expected values corresponding to each type. If it is determined that there is a current target expected value less than the second preset threshold among the current target expected values corresponding to each type, then the current query statement set corresponding to the current target expected value less than the second preset threshold will be deleted.
[0144] Optionally, the second preset threshold can be 0.2M, or other thresholds, which can be set according to the actual situation.
[0145] S105. Based on the current execution frequency peak value corresponding to each type, determine the current target execution frequency peak value corresponding to each type.
[0146] Optionally, in another embodiment of this application, one specific implementation of step S105 is as follows: Figure 6 As shown, it includes the following steps:
[0147] S601. Based on the common feature values of the current query statement sets of each type, find out if there are peak execution frequencies of historical targets corresponding to each type.
[0148] It should be noted that the specific implementation of step S602 can be referred to step S401 accordingly, and will not be repeated here.
[0149] It should also be noted that if no historical execution frequency peak is found for each type, it indicates that each type may be being calculated for the first time, or the previous calculation result may not be valid. Therefore, step S602 needs to be executed. If historical execution frequency peaks are found for each type, it indicates that the current execution frequency peak for each type needs to be integrated and referenced with the previous calculation result. Therefore, step S603 needs to be executed.
[0150] S602. Determine the current execution frequency peak value corresponding to each type as the current target execution frequency peak value corresponding to each type.
[0151] S603. The current execution frequency peak value corresponding to each type and its corresponding historical target execution frequency peak value are weighted and summed to obtain the current target execution frequency peak value corresponding to each type.
[0152] Specifically, the calculation expression for the integrated target execution frequency peaks for each type is obtained by weighted summation of the current execution frequency peaks for each type calculated based on S104 and their corresponding historical target execution frequency peaks:
[0153] Y = y1*f1 + y2*f2 (when y1 > y2)
[0154] Y = y1*f2 + y2*f1 (when y1 <y2)
[0155] In particular, when y1 = 0, Y = y2.
[0156] Where y1 is the historical target execution frequency peak value corresponding to each type (0 if it does not exist), y2 is the current execution frequency peak value corresponding to each type (0 if it does not exist), Y is the target execution frequency peak value corresponding to each type after integration, f1 and f2 are weights, and f1>f2, f1+f2=1. The weights can be set according to the requirements, but the condition must be met, that is, this calculation method tends to retain the larger execution frequency peak value obtained from the two sampling calculations.
[0157] Then, when y2 = 0, Y = y1 * f1, which means that no corresponding query statement was found in the current query statement set of each type. Therefore, the target execution frequency peak Y corresponding to each type needs to be assigned as the historical target execution frequency peak corresponding to each type multiplied by a weight less than 1.
[0158] S106. Based on the expected value of the current target for each type and the peak value of the current target execution frequency, calculate the peak memory usage for each type.
[0159] Specifically, based on the current target expectation value corresponding to each type calculated in step S103 and the current target execution frequency peak value corresponding to each type calculated in step S105, the memory usage peak value corresponding to each type can be calculated by the target algorithm, where the target algorithm is: memory usage peak value = current target expectation value × current target execution frequency peak value.
[0160] S107. Calculate the total peak memory usage within the target time period based on the peak memory usage for each type and its corresponding current target expectation value.
[0161] Specifically, based on the peak memory usage for each type and its corresponding current target expectation, the total peak memory usage can be calculated using a preset algorithm. The preset algorithm is as follows:
[0162] Total=f1+(f2+avg2) / 2+(f3+avg3) / 2+(f4+avg4) / 2+…
[0163] Where f1 represents the maximum value of the peak memory usage of all current query statement sets, f2, f3, f4... represent the peak memory usage of all other current query statement sets except the one corresponding to f1, and avg2, avg3, and avg4 represent the expected memory usage of the current query statement sets corresponding to f2, f3, and f4, respectively.
[0164] S108. Determine whether the total peak value is not less than the first preset threshold.
[0165] It should be noted that the total peak memory usage calculated in step S107 may not have a memory overflow situation, so there is no need to send a memory overflow warning. Therefore, in order to know whether the total peak has reached the memory overflow value, it is necessary to further determine the total peak, that is, to determine whether the total peak is greater than the first preset threshold. If it is determined that the total peak is greater than the first preset threshold, it means that the total peak has reached the memory overflow value, and step S109 needs to be executed. If it is determined that the total peak is less than the first preset threshold, then the memory usage of each type continues to be monitored.
[0166] Optionally, the first preset threshold can be 2G, or other thresholds. A suitable threshold can be set according to the actual memory situation of the system server.
[0167] S109. Feedback the early warning information to the front end.
[0168] Specifically, when the total peak value is not less than the first preset threshold, an early warning message needs to be sent to the front end in a timely manner. The early warning message is mainly used to inform staff that there is a memory overflow and to clean up the memory in time to avoid affecting the processing efficiency of the work.
[0169] This application provides a memory overflow early warning method. It obtains a set of current query statements of various types within a target time period, calculates the expected memory usage for each type of current query statement set, and then determines the current target expected value for each type. Next, it calculates the peak execution frequency of each type of current query statement set, and then determines the current target execution frequency peak for each type. Finally, it calculates the peak memory usage for each type based on the current target expected value and its corresponding peak execution frequency, and then calculates the total peak memory usage within the target time period based on the peak memory usage and the corresponding current target expected value. If the total peak value exceeds a first preset threshold, an early warning message is sent to the front end. This method solves the problem of a high false alarm rate for memory overflows due to neglecting the differences in executed statements, and also addresses the problem of excessive system resource consumption due to not systematically considering the regularity of calculations.
[0170] Another embodiment of this application provides a memory overflow early warning device, such as... Figure 7 As shown, it includes the following units:
[0171] The first acquisition unit 701 is used to acquire the current query statement set of each type within the target time period.
[0172] The first calculation unit 702 is used to calculate the expected value of memory occupied by the current query statement set of each type, and obtain the current expected value corresponding to each type.
[0173] The first determining unit 703 is used to determine the current target expected value for each type based on the current expected value for each type.
[0174] The second calculation unit 704 is used to calculate the peak execution frequency of the current query statement set of each type, and obtain the peak execution frequency of each type.
[0175] The second determining unit 705 is used to determine the current target execution frequency peak value corresponding to each type based on the current execution frequency peak value corresponding to each type.
[0176] The third calculation unit 706 is used to calculate the peak memory usage for each type based on the expected value of the current target and the peak execution frequency of the current target corresponding to each type.
[0177] The fourth calculation unit 707 is used to calculate the total peak memory usage within the target time period based on the peak memory usage corresponding to each type and its corresponding current target expected value.
[0178] Feedback unit 708 is used to send a warning message to the front end if the total peak value is not less than the first preset threshold.
[0179] It should be noted that the specific working process of the above-mentioned units in the embodiments of this application can be referred to steps S101 to S109 in the above method embodiments, and will not be repeated here.
[0180] Optionally, in another embodiment of this application, a memory overflow early warning device includes a first acquisition unit comprising:
[0181] The second acquisition unit is used to acquire each current query statement within the target time period.
[0182] The classification unit is used to classify each current query statement according to preset rules, so as to obtain a set of current query statements of each type.
[0183] The identification unit is used to identify the set of current query statements for each type by using the common feature values of the current query statements for each type.
[0184] It should be noted that the specific working process of each unit provided in the above embodiments of this application can be referred to the corresponding steps in the above method embodiments, and will not be repeated here.
[0185] Optionally, in another embodiment of this application, a memory overflow early warning device includes a first determining unit comprising:
[0186] The first search unit is used to search for the existence of historical target expected values corresponding to each type based on the common feature values of the current query statement sets of each type.
[0187] The third determining unit is used to determine the current expected value corresponding to each type as the current expected value corresponding to each type if no historical target expected value is found for each type.
[0188] The first weighted summation unit is used to sum the current expected value and its corresponding historical expected value for each type if a historical expected value for each type is found, in order to obtain the current expected value for each type.
[0189] It should be noted that the specific working process of each unit provided in the above embodiments of this application can be referred to the corresponding steps in the above method embodiments, and will not be repeated here.
[0190] Optionally, in another embodiment of the present application, a memory overflow early warning device includes a second determining unit comprising:
[0191] The second search unit is used to search for the existence of historical target execution frequency peaks corresponding to each type based on the common feature values of the current query statement sets of each type.
[0192] The fourth determining unit is used to determine the current execution frequency peak corresponding to each type as the current target execution frequency peak corresponding to each type if no historical execution frequency peak corresponding to each type is found.
[0193] The second weighted summation unit is used to sum the current execution frequency peak corresponding to each type and its corresponding historical target execution frequency peak if the historical target execution frequency peak corresponding to each type is found, so as to obtain the current target execution frequency peak corresponding to each type.
[0194] It should be noted that the specific working process of each unit provided in the above embodiments of this application can be referred to the corresponding steps in the above method embodiments, and will not be repeated here.
[0195] Optionally, in another embodiment of this application, a memory overflow early warning device includes a first computing unit comprising:
[0196] The third acquisition unit is used to obtain the memory usage value of each current query statement of each type through aspect-based methods.
[0197] The fifth calculation unit is used to calculate the average memory usage of each current query statement of each type, and obtain the average value corresponding to each type.
[0198] The fourth acquisition unit is used to acquire the total number of samples of each type of current query statement set within the target time period and the number of time units within the target time period.
[0199] The sixth calculation unit is used to calculate the execution frequency of the current query statement set of each type based on the total number of samples and the number of time units within the target time period, and to obtain the execution frequency corresponding to each type.
[0200] The seventh calculation unit is used to calculate the expected memory occupied by the current query statement set of each type based on the average value and execution frequency of each type, and to obtain the current expected value for each type.
[0201] It should be noted that the specific working process of each unit provided in the above embodiments of this application can be referred to the corresponding steps in the above method embodiments, and will not be repeated here.
[0202] Optionally, in another embodiment of this application, a memory overflow early warning device includes a second computing unit comprising:
[0203] The fifth acquisition unit is used to acquire the execution timestamp of each current query statement of each type.
[0204] The eighth calculation unit is used to calculate the maximum number of times each current query statement is executed within each time unit of the target time period based on the execution timestamp of each current query statement of each type, and obtain the maximum number of times corresponding to each type.
[0205] The fifth determining unit is used to determine the maximum number of times corresponding to each type as the current execution frequency peak value corresponding to each type.
[0206] It should be noted that the specific working process of each unit provided in the above embodiments of this application can be referred to the corresponding steps in the above method embodiments, and will not be repeated here.
[0207] Optionally, another embodiment of this application provides a memory overflow early warning device, which further includes:
[0208] The judgment unit is used to determine whether there is a current target expected value that is less than a second preset threshold among the current target expected values corresponding to each type.
[0209] The deletion unit is used to delete the set of current query statements corresponding to the types of current target expected values that are less than the second preset threshold if it is determined that there are current target expected values less than the second preset threshold among the current target expected values of each type.
[0210] It should be noted that the specific working process of each unit provided in the above embodiments of this application can be referred to the corresponding steps in the above method embodiments, and will not be repeated here.
[0211] Another embodiment of this application provides an electronic device, such as... Figure 8 As shown, it includes:
[0212] Memory 801 and processor 802.
[0213] Among them, memory 801 is used to store programs.
[0214] The processor 802 is used to execute a program, which, when executed, is specifically used to implement a memory overflow warning method as provided in any of the above embodiments.
[0215] Another embodiment of this application provides a computer storage medium for storing a computer program, which, when executed, implements a memory overflow warning method as provided in any of the above embodiments.
[0216] Computer storage media, including both permanent and non-permanent, removable and non-removable media, can store information using any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.
[0217] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.
[0218] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A method for early warning of memory overflow, characterized in that, include: Retrieve the collection of current query statements of various types within the target time period; Calculate the expected memory usage of the current query statement set for each of the aforementioned types to obtain the current expected value for each type. Based on the current expected value corresponding to each of the aforementioned types, determine the current target expected value corresponding to each of the aforementioned types; Calculate the peak execution frequency of the current query statement set for each type to obtain the peak execution frequency for each type. Based on the current execution frequency peak value corresponding to each of the aforementioned types, determine the current target execution frequency peak value corresponding to each of the aforementioned types; Based on the current target expected value and the corresponding current target execution frequency peak value of each type, calculate the peak memory usage of each type; Based on the peak memory usage corresponding to each type and its corresponding current target expected value, calculate the total peak memory usage within the target time period; If the total peak value is not less than the first preset threshold, then a warning message is sent to the front end.
2. The method according to claim 1, characterized in that, The process of obtaining the set of current query statements of various types within the target time period includes: Retrieve all current query statements within the target time period; According to preset rules, each current query statement is classified to obtain a set of current query statements of each type; For each of the aforementioned types, the set of current query statements for that type is identified using the common feature values of the current query statements for that type.
3. The method according to claim 2, characterized in that, The step of determining the current target expected value for each type based on the current expected value for each type includes: Based on the common feature values of the current query statement sets of each type, check whether there are historical target expected values corresponding to each type; If no historical target expected value is found for each of the aforementioned types, then the current expected value for each of the aforementioned types is determined as the current target expected value for each of the aforementioned types. If historical target expected values are found for each of the aforementioned types, then the current expected value for each type and its corresponding historical target expected value are weighted and summed to obtain the current target expected value for each type.
4. The method according to claim 2, characterized in that, The step of determining the current target execution frequency peak value corresponding to each of the aforementioned types based on the current execution frequency peak value includes: Based on the common feature values of the current query statement sets of each type, search for whether there are historical target execution frequency peaks corresponding to each type; If no historical target execution frequency peak is found for each of the aforementioned types, then the current execution frequency peak for each of the aforementioned types is determined as the current target execution frequency peak for each of the aforementioned types. If historical target execution frequency peaks corresponding to each of the aforementioned types are found, then the current target execution frequency peaks corresponding to each of the aforementioned types and their corresponding historical target execution frequency peaks are weighted and summed to obtain the current target execution frequency peaks corresponding to each of the aforementioned types.
5. The method according to claim 1, characterized in that, The step of calculating the expected memory usage of the current query statement set for each type to obtain the current expected value for each type includes: The memory usage of each current query statement of each type is obtained by aspect-based approach; Calculate the average memory usage of each current query statement for each type to obtain the average value corresponding to each type; Obtain the total number of samples of the current query statement set of each type within the target time period and the number of time units within the target time period; Based on the total number of samples and the number of time units within the target time period, the execution frequency of the current query statement set of each type is calculated to obtain the execution frequency corresponding to each type. Based on the average value and execution frequency of each type, the expected memory usage of the current query statement set for each type is calculated using the target algorithm, thus obtaining the current expected value for each type.
6. The method according to claim 1, characterized in that, The step of calculating the peak execution frequency of the current query statement set for each type to obtain the peak execution frequency corresponding to each type includes: Obtain the execution timestamp of each of the current query statements of each of the aforementioned types; Based on the execution timestamp of each current query statement of each type, calculate the maximum number of times each current query statement of each type is executed in each time unit of the target time period, and obtain the maximum number of times corresponding to each type; The maximum number of times corresponding to each type is determined as the current execution frequency peak value corresponding to each type.
7. The method according to claim 1, characterized in that, Before calculating the peak execution frequency of the current query statement set for each type to obtain the peak execution frequency corresponding to each type, the method further includes: Determine whether there is a current target expected value that is less than a second preset threshold among the current target expected values corresponding to each type; If it is determined that there is a current target expected value less than the second preset threshold among the current target expected values corresponding to each of the types, then the set of current query statements corresponding to the types with current target expected values less than the second preset threshold will be deleted.
8. A memory overflow early warning device, characterized in that, include: The first acquisition unit is used to acquire a set of current query statements of various types within the target time period; The first calculation unit is used to calculate the expected value of memory occupied by the current query statement set of each type, and obtain the current expected value corresponding to each type; The first determining unit is used to determine the current target expected value corresponding to each of the types based on the current expected value corresponding to each type; The second calculation unit is used to calculate the peak execution frequency of the current query statement set for each type, and obtain the peak execution frequency corresponding to each type. The second determining unit is used to determine the current target execution frequency peak value corresponding to each of the types based on the current execution frequency peak value corresponding to each type. The third calculation unit is used to calculate the peak memory usage corresponding to each type based on the current target expected value and the corresponding current target execution frequency peak value of each type. The fourth calculation unit is used to calculate the total peak memory usage within the target time period based on the peak memory usage corresponding to each type and its corresponding current target expected value. The feedback unit is used to send a warning message to the front end if the total peak value is not less than a first preset threshold.
9. An electronic device, characterized in that, include: Memory and processor; The memory is used to store programs; The processor is used to execute the program, which, when executed, is specifically used to implement a memory overflow warning method as described in any one of claims 1 to 7.
10. A computer storage medium, characterized in that, Used to store a computer program, which, when executed, implements a memory overflow warning method as described in any one of claims 1 to 7.
Citation Information
Patent Citations
Memory overflow detection method and system
CN111679926A
Memory overflow processing method and device, electronic equipment and storage medium
CN112965851A