Resource estimation method and program product

Abstract

The invention provides a resource estimation method and a program product. A memory resource occupancy estimation value is defined as the optimal estimation of a memory resource occupancy theoretical value and a memory resource occupancy measured value through Bayesian fusion. The memory resource occupancy pre-estimated value not only comprises the optimal pre-estimated value of memory resource occupancy, but also completely retains uncertainty information of estimation. Compared with simply using the theoretical value of memory resource occupancy, the Bayesian fusion result effectively corrects the deviation caused by unmodeled factors in the theoretical value of memory resource occupancy by absorbing the actual operation observation data, so that the estimated value of memory resource occupancy better conforms to the real environment performance. Therefore, the memory resource occupancy pre-estimation value which is close to reality and has statistical stability is provided, finally, pre-estimation of resources corresponding to the calculation task is achieved more lightly, flexibly and accurately, and then scheduling of the calculation task can be completed quantitatively.

Description

Technical Field

[0001] This application relates to the field of data computing, and more particularly, to a resource estimation method and a program product. Background Art

[0002] In the financial field, there are a large number of data analysis requirements, and the analysis and calculation scenarios are complex and changeable. For example, the analysis objects, dimensions, calculation calibers, etc. all have dynamic change characteristics. At the same time, with the growth of business data, the scale and concurrency of the analysis and calculation data will also show randomness. The data analysis system will simultaneously process high-concurrency low-latency calculations and large-scale data high-latency calculations. Therefore, higher requirements are put forward for the memory management of this system.

[0003] In the prior art, usually through program optimization, parameter tuning, request rate limiting and other methods, to reduce the risk of out-of-memory (OOM) during the calculation process.

[0004] However, the optimization mechanism of the prior art often adopts a fixed configuration based on experience, which has poor adaptability to dynamic scenarios and random sudden changes, and often requires manual intervention, thus reducing the efficiency and resource utilization rate. Summary of the Invention

[0005] The purpose of this application is to provide a resource estimation method and a program product, which are used to provide a memory resource occupancy predicted value that is both close to the actual situation and has statistical stability, and then can quantitatively complete the scheduling of calculation tasks.

[0006] To achieve the above purpose, the technical solutions adopted in the embodiments of this application are as follows: In a first aspect, an embodiment of this application provides a resource estimation method, including: Estimate the theoretical value of the memory resource occupancy of any operator during the analysis and calculation process; the operator corresponds to the smallest calculation unit for data during the analysis and calculation process; According to the measured use case and the corresponding execution parameter sequence, estimate the measured value of the memory resource occupancy of the corresponding operator during the analysis and calculation process; Perform Bayesian fusion on the theoretical value of the memory resource occupancy and the corresponding measured value of the memory resource occupancy to obtain the predicted value of the memory resource occupancy of the operator; When a target calculation request is obtained, according to the target execution parameter sequence of the target calculation request, at least one target operator corresponding to the target calculation request, and the target memory resource occupancy predicted values corresponding to all the target operators, determine the peak value of the memory resource occupancy corresponding to the target calculation request; Determine whether the current memory resource capacity meets the peak memory resource occupancy. If it meets, call the computing engine to analyze and calculate the target computing request; If it does not meet, put the target computing request into the buffer queue to wait.

[0007] Optionally, the step of estimating the theoretical value of the memory resource occupancy of any operator during the analysis and calculation includes: Recursively traverse the abstract syntax tree corresponding to each operator to obtain the recursive level information corresponding to each operator; Obtain the theoretical value of the resource occupancy and the number of objects of various types corresponding to each operator; Determine the theoretical value of the memory resource occupancy according to the recursive level information, the theoretical value of the resource occupancy of various types of objects, and the number of objects.

[0008] Optionally, the step of estimating the measured value of the memory resource occupancy of the corresponding operator during the analysis and calculation according to the measured use case and the corresponding execution parameter sequence includes: Through the data scale ladder of the measured use case and the corresponding execution parameter sequence, test and obtain the measured peak value of the memory resource occupancy of the measured use case; Taking the data scale as the independent variable and the measured peak value of the memory resource occupancy as the dependent variable, obtain the corresponding measured value of the memory resource occupancy through a linear regression model.

[0009] Optionally, the step of performing Bayesian fusion on the theoretical value of the memory resource occupancy and the corresponding measured value of the memory resource occupancy to obtain the predicted value of the memory resource occupancy of the operator includes: Perform Bayesian fusion on the theoretical value of the memory resource occupancy and the corresponding measured value of the memory resource occupancy to obtain the posterior distribution; Take the mean of the posterior distribution to obtain the predicted value of the memory resource occupancy of the operator.

[0010] Optionally, after the step of performing Bayesian fusion on the theoretical value of the memory resource occupancy and the corresponding measured value of the memory resource occupancy to obtain the predicted value of the memory resource occupancy of the operator, it further includes: Construct the operator metadata of each operator by combining the predicted value of the memory resource occupancy of each operator, the minimum data volume for each operator to run, and the basic operator information corresponding to each operator; Among all the operator metadata, determine the operators corresponding to each data analysis directed acyclic graph DAG; Determine the corresponding critical path according to the operators corresponding to each piece of the data analysis DAG; the critical path represents the path with the largest memory resource occupancy in the corresponding data analysis DAG; each path is composed of multiple operators in the data analysis DAG; Determine the estimated peak value of the memory resource occupancy of the data analysis DAG corresponding to the data analysis DAG according to all the operators included in the critical path.

[0011] Optionally, when a target calculation request is obtained, the step of determining the peak value of the memory resource occupancy corresponding to the target calculation request according to the target execution parameter sequence of the target calculation request, at least one target operator corresponding to the target calculation request, and the estimated values of the target memory resource occupancies corresponding to all the target operators includes: When a target calculation request is obtained, determine the peak value of the memory resource occupancy corresponding to the target calculation request according to the target execution parameter sequence of the target calculation request, the data analysis DAG corresponding to the target calculation request, and the estimated peak value of the memory resource occupancy of the data analysis DAG.

[0012] Optionally, it further includes: Divide the heap memory resources into an initial area, a running area, an available area, and a fault-tolerant area; wherein, the initial area is the used heap memory resources after the system starts; the running area is the total estimated value of the resource occupancy of the target calculation request; the available area is the logically available heap memory resources of the system; the fault-tolerant area is the heap memory resources multiplied by a fault-tolerant coefficient, and the fault-tolerant coefficient is a constant ratio.

[0013] Optionally, the step of determining whether the current memory resource capacity meets the peak value of the memory resource occupancy includes: Compare the available area with the peak value of the memory resource occupancy; If the available area is greater than the peak value of the memory resource occupancy, call the computing engine to perform analysis and calculation on the target calculation request; If the available area is less than or equal to the peak value of the memory resource occupancy, put the target calculation request into the buffer queue to wait.

[0014] In a second aspect, an embodiment of the present application provides a resource estimation device, including: a resource estimation model and a task manager; The resource estimation model is used to estimate the theoretical value of the memory resource occupancy of any operator during the analysis and calculation process; the operator corresponds to the smallest calculation unit for data during the analysis and calculation process; estimate the measured value of the memory resource occupancy of the corresponding operator during the analysis and calculation process according to the measured use case and the corresponding execution parameter sequence; perform Bayesian fusion on the theoretical value of the memory resource occupancy and the corresponding measured value of the memory resource occupancy to obtain the estimated value of the memory resource occupancy of the operator; The task manager is used to, when obtaining a target computing request, determine the peak memory resource occupancy corresponding to the target computing request according to the target execution parameter sequence of the target computing request, at least one target operator corresponding to the target computing request, and the predicted values of the target memory resource occupancies corresponding to all the target operators; determine whether the current memory resource capacity meets the peak memory resource occupancy, and if it does, call the computing engine to perform analysis and calculation on the target computing request; if it does not, put the target computing request into a buffer queue to wait.

[0015] Optionally, the resource prediction model is specifically used for: Recursively traverse the abstract syntax tree corresponding to each operator to obtain the recursive level information corresponding to each operator; obtain the theoretical resource occupancies and the number of objects of various types corresponding to each operator; and determine the theoretical memory resource occupancy according to the recursive level information, the theoretical resource occupancies of various types of objects, and the number of objects.

[0016] Optionally, the resource prediction model is specifically used for: Through the data scale ladder of the measured use cases and the corresponding execution parameter sequences, test and obtain the measured peak memory resource occupancy of the measured use cases; use the data scale as the independent variable and the measured peak memory resource occupancy as the dependent variable, and obtain the corresponding measured memory resource occupancy through a linear regression model.

[0017] Optionally, it further includes: a metadata manager; The metadata manager is used to form the operator metadata of each operator by the predicted value of the memory resource occupancy of each operator, the minimum data volume for each operator to run, and the basic operator information corresponding to each operator; determine the operators corresponding to each data analysis directed acyclic graph (DAG) among all the operator metadata; determine the corresponding critical path according to the operators corresponding to each data analysis DAG; the critical path represents the path with the largest memory resource occupancy in the corresponding data analysis DAG; each path is composed of multiple operators in the data analysis DAG; and determine the predicted peak memory resource occupancy of the data analysis DAG corresponding to the data analysis DAG according to all the operators included in the critical path.

[0018] Optionally, the task manager is specifically used for: When obtaining a target computing request, determine the peak memory resource occupancy corresponding to the target computing request according to the target execution parameter sequence of the target computing request, the data analysis DAG corresponding to the target computing request, and the predicted peak memory resource occupancy of the data analysis DAG.

[0019] Optionally, it further includes: a resource manager; The resource manager is configured to divide the heap memory resources into an initial area, a running area, an available area, and a fault-tolerant area; wherein, the initial area is the used heap memory resources after the system starts; the running area is the total estimated value of the resource occupation of the target computing request; the available area is the logically available heap memory resources of the system; the fault-tolerant area is the heap memory resources multiplied by a fault-tolerant coefficient, and the fault-tolerant coefficient is a constant ratio.

[0020] Optionally, the resource manager is specifically configured to: Compare the available area with the peak value of the memory resource occupation; if the available area is greater than the peak value of the memory resource occupation, call the computing engine to analyze and calculate the target computing request; if the available area is less than or equal to the peak value of the memory resource occupation, put the target computing request into the buffer queue to wait.

[0021] In a third aspect, an embodiment of the present application provides an electronic device, including: A memory for storing one or more programs; A processor; When the one or more programs are executed by the processor, the method described in any one of the above first aspects is implemented.

[0022] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored, and characterized in that when the computer program is executed by a processor, the method described in any one of the above first aspects is implemented.

[0023] In a fifth aspect, an embodiment of the present application provides a program product, which when executed by a processor, implements the method described in any one of the above first aspects.

[0024] Compared with the prior art, for a resource estimation method and a program product provided by an embodiment of the present application, the predicted value of the memory resource occupation is defined as the optimal estimation obtained by Bayesian fusion of the theoretical value of the memory resource occupation and the measured value of the memory resource occupation. This predicted value of the memory resource occupation not only includes the best predicted value of the memory resource occupation, but also completely retains the uncertainty information of the estimation. Compared with simply using the theoretical value of the memory resource occupation, the result of this Bayesian fusion effectively corrects the deviation caused by the unmodeled factors in the theoretical value of the memory resource occupation by absorbing the actual operation observation data (measured use cases), making the predicted value of the memory resource occupation more in line with the actual environment performance. Thus, it provides a predicted value of the memory resource occupation that is both close to the actual situation and has statistical stability, and finally more lightly, flexibly, and accurately realizes the estimation of the resources corresponding to the computing task, and further can quantitatively complete the scheduling of the computing task.

[0025] To make the above objects, features, and advantages of the present application more obvious and understandable, the following provides preferred embodiments in conjunction with the accompanying drawings and describes them in detail as follows. Description of the Drawings

[0026] To more clearly illustrate the technical solutions of the embodiments of the present application, the following briefly introduces the drawings required for the embodiments. It should be understood that the following drawings only show some embodiments of the present application and should not be regarded as limiting the scope. For those of ordinary skill in the art, without creative efforts, other related drawings can also be obtained based on these drawings.

[0027] Figure 1 Schematic diagram of the OOM risk control system for the prior art; Figure 2 Schematic flowchart of a resource estimation method provided by an embodiment of the present invention; Figure 3 Schematic flowchart of another resource estimation method provided by an embodiment of the present invention; Figure 4 Schematic flowchart of another resource estimation method provided by an embodiment of the present invention; Figure 5 Schematic flowchart of another resource estimation method provided by an embodiment of the present invention; Figure 6 Schematic diagram of a data analysis DAG provided by an embodiment of the present invention; Figure 7 Schematic flowchart of another resource estimation method provided by an embodiment of the present invention; Figure 8 Schematic diagram of a resource estimation device provided by an embodiment of the present invention; Figure 9 Schematic diagram of another resource estimation device provided by an embodiment of the present invention; Figure 10 Schematic diagram of another resource estimation device provided by an embodiment of the present invention; Figure 11 Schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. Detailed Embodiments

[0028] To make the objectives, technical solutions, and advantages of the embodiments of the present application clearer, the following will clearly and completely describe the technical solutions in the embodiments of the present application in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are some, but not all, of the embodiments of the present application. Usually, the components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various different configurations.

[0029] Accordingly, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the claimed present application, but merely represents selected embodiments of the present application. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in the present application without creative efforts shall fall within the scope of protection of the present application.

[0030] It should be noted that like reference numerals and letters denote like items in the following drawings. Therefore, once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings. At the same time, in the description of the present application, terms such as "first", "second", etc. are only used for distinguishing descriptions and cannot be understood as indicating or implying relative importance.

[0031] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements not only includes those elements, but also includes other elements not expressly listed, or also includes elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "comprising a..." does not exclude the presence of additional identical elements in the process, method, article or device comprising the element.

[0032] In the prior art, in order to reduce the risk of OOM during the data analysis process, it is usually divided into two stages, that is, the mechanism for deploying risk control during the "pre-run" and "runtime" of the calculation process of data analysis. Specifically, Figure 1 For the schematic diagram of the OOM risk control system of the prior art, see Figure 1 , before running, program optimization and static parameter tuning - Java Virtual Machine (JVM for short) are usually used. Among them, "static parameter tuning JVM" refers to optimizing and adjusting the static parameters of the JVM to reduce the risk of memory-related problems such as out-of-memory errors during the operation of the JVM, thereby improving the stability and performance of the JVM.

[0033] Furthermore, a request rate limiting scheme is set during the "runtime", and this request rate limiting scheme can be implemented by setting corresponding thresholds for the upper limit of the use of memory resources. For example, Figure 1As shown in the figure, when the client initiates a calculation request, the "request interceptor" is used to confirm whether the usage rate of the current memory resources exceeds a "threshold". If the usage rate is greater than the threshold, it means "insufficient resources", and the calculation request is cached in the waiting queue so that when the usage rate of the subsequent memory resources is lower than the threshold, the data analysis corresponding to the calculation request is executed by the service processor.

[0034] If the usage rate is less than the threshold, it means "sufficient resources", and the data analysis corresponding to the calculation request is executed by the service processor.

[0035] In this solution, the resource manager realizes the flow control of calculation requests by setting the threshold and monitoring the memory resources.

[0036] However, the above method has the following defects: First, static technical solutions such as parameter tuning and optimization all refer to conventional business scenarios and fix a set of configurations according to empirical values, which cannot adapt to various dynamic scenarios and random sudden changes. Manual intervention and adjustment are required after an OOM occurs.

[0037] Second, the request flow control scheme needs to set a monitoring threshold for memory resources, and it is impossible to accurately estimate and distinguish between high-resource requests and low-resource requests, resulting in a decrease in resource utilization rate. Some low-resource calculation requests are mistakenly put into the blocking queue, and there is a possibility that large-scale data calculation requests directly break through the threshold and cause an OOM.

[0038] Third, as heavyweight computing engines, Spark or Flink require a large amount of additional hardware and operation and maintenance costs, and the feasibility is relatively low. Also, due to the inability to estimate resource consumption during the calculation process, algorithm complexity explosion, Spark partition data skew, Flink backpressure failure and data accumulation, etc. can also directly cause an OOM.

[0039] To solve the above technical problems, the present application provides a resource estimation mechanism, which uses the theoretical value of memory resource occupancy representing the theoretical expectation and the measured value of memory resource occupancy representing the measured data to dynamically estimate the expected resource occupancy of the calculation request, and on this basis, realizes the calculation task scheduling of data analysis. Thus, the problems in the prior art such as inability to estimate or large estimation deviation, manual intervention and optimization required after an OOM occurs, and introduction of heavyweight middleware are solved.

[0040] Optionally, the core process of the present application is exemplarily described below. Specifically, Figure 2 is a schematic flowchart of a resource estimation method provided by an embodiment of the present invention. Refer to Figure 2 and the method includes: Step 102, estimate the theoretical value of the memory resource occupancy of any operator during the analysis and calculation process.

[0041] Among them, the operator corresponds to the smallest calculation unit for data in the analysis and calculation process.

[0042] Step 103: According to the measured use case and the corresponding execution parameter sequence, estimate the measured value of the memory resource occupancy of the corresponding operator in the analysis and calculation process.

[0043] Among them, the measured use case can include a batch or multiple batches of actual operation observation data.

[0044] Step 104: Perform Bayesian fusion on the theoretical value of memory resource occupancy and the corresponding measured value of memory resource occupancy to obtain the estimated value of the memory resource occupancy of the operator.

[0045] Among them, the Bayesian Fusion can combine prior knowledge (such as the theoretical value of memory resource occupancy) with new observation data (such as the measured value of memory resource occupancy) to obtain a result closer to the actual situation after analysis, that is, the estimated value of memory resource occupancy.

[0046] Step 109: When a target calculation request is obtained, determine the peak value of the memory resource occupancy corresponding to the target calculation request according to the target execution parameter sequence of the target calculation request, at least one target operator corresponding to the target calculation request, and the estimated value of the target memory resource occupancy corresponding to all target operators.

[0047] Among them, the target operator is one or more of all the operators obtained through Step 104.

[0048] Step 110: Determine whether the current memory resource capacity meets the peak value of the memory resource occupancy.

[0049] Specifically, if it is satisfied, execute Step 111; if not, execute Step 112.

[0050] Step 111: Call the computing engine to perform analysis and calculation on the target calculation request.

[0051] Step 112: Put the target calculation request into the buffer queue and wait.

[0052] Optionally, Step 112 can return to execute Step 110. When the current memory resource capacity is updated, Step 110 can be rejudged until the analysis and calculation are completed through Step 111 when it is satisfied.

[0053] The resource estimation method provided by the embodiments of this application defines the predicted value of memory resource occupancy as the optimal estimate obtained by Bayesian fusion of the theoretical value of memory resource occupancy and the measured value of memory resource occupancy. This predicted value of memory resource occupancy not only includes the best predicted value of memory resource occupancy but also completely retains the uncertainty information of the estimate. Compared with simply using the theoretical value of memory resource occupancy, the result of this Bayesian fusion effectively corrects the deviation caused by unmodeled factors in the theoretical value of memory resource occupancy by absorbing actual running observation data (measured use cases), making the predicted value of memory resource occupancy more in line with the actual environment performance. Thus, it provides a predicted value of memory resource occupancy that is both close to reality and has statistical stability, and finally realizes the estimation of the corresponding resources of the computing task more lightly, flexibly, and accurately, and then can quantitatively complete the scheduling of the computing task. In addition, since the resource estimation method provided by the embodiments of this application can be embedded and integrated into the application system, thus realizing integrated deployment with the application system, therefore, no additional hardware and operation and maintenance costs are required.

[0054] In a possible implementation manner, a possible implementation manner of the "theoretical calculation method" is provided to estimate the theoretical value of memory resource occupancy of each operator during the analysis and calculation process. Optionally, this "theoretical calculation method" uses an "automatic parsing framework" to recursively traverse the code involved in the operator to obtain all object and structure information at each level of the code corresponding to the operator, and then estimates the resource occupancy of all objects to obtain the theoretical value of memory resource occupancy of the operator. Specifically, on the basis of Figure 2 Figure 3 This is a schematic flowchart of another resource estimation method provided by the embodiments of the present invention. Refer to Figure 3 , step 102 includes: Step 102-1: Recursively traverse the abstract syntax tree corresponding to each operator to obtain the recursive level information corresponding to each operator.

[0055] Among them, for each operator, its corresponding code can be represented by an abstract syntax tree, and then this abstract syntax tree can contain objects at different levels. And the size of all objects included in the operator determines the theoretical value of memory resource occupancy of the operator. Therefore, only by recursively traversing to obtain the recursive level information corresponding to the operator can all objects and their attribute structure information at each level be obtained.

[0056] Optionally, for the mechanism of recursive traversal, its termination condition can be: all calls to functions and methods at each level of the abstract syntax tree have been completed, then the recursive traversal terminates.

[0057] Optionally, this all object and its attribute structure information can also be persistently recorded.

[0058] Step 102-2: Obtain the theoretical resource occupation values and the number of objects corresponding to each operator for various types of objects.

[0059] Among them, the theoretical resource occupation value of the object represents the object size. The expression of the object size can be: Size of a single object = object header + instance data + alignment padding Specifically, the object header size depends on the JVM implementation and whether compressed pointers are enabled (usually 12 or 16 bytes); the instance data size is the total space actually occupied by all fields; alignment padding size = (8 - (object header size + instance data size) % 8) % 8.

[0060] For example, assume a 64-bit JVM with compressed pointers enabled (object header = 12 bytes).

[0061] Calculate: Object header = 12 bytes Instance data = 8(id) + 4(name) + 8(price) = 20 bytes Alignment padding = (8 - (12 + 20) % 8) % 8 = (8 - 32 % 8) % 8 = (8 - 0) % 8 = 0 bytes (because 32 is already a multiple of 8).

[0062] The following provides a Java example of an object: public class Product { long id; / / 8 bytes string name; / / 4 bytes (reference) double price; / / 8 bytes } Among them, long id, string name, and double price can be the attribute structure information of the object. Obviously, for different examples, the attribute structure information of the object is different and is not limited here.

[0063] Step 102-3: Determine the theoretical memory resource occupation value according to the recursive level information, the theoretical resource occupation values of various types of objects, and the number of objects.

[0064] Optionally, most of the data structures can be object collections or object arrays, and there is an approximately linear relationship between the data scale and the memory resource occupation. Take the maximum value (process peak value) of each recursive level. Therefore, the expression of the theoretical memory resource occupation value can be: Mtheory = Max(∑(Size of single object * Number of objects) of recursive hierarchical information) In summary, the present application uses the theoretical calculation method to complete the estimation of the theoretical value of the memory resource occupancy during the analysis and calculation of any operator. Optionally, the theoretical value of the memory resource occupancy is estimated based on the minimum data volume during the operation of the corresponding operator.

[0065] In a possible implementation, in order to better estimate the actual measured value of the memory resource occupancy of each operator during the analysis and calculation, the "actual measurement regression method" can be used to achieve the specific estimation. Specifically, on the basis of Figure 2 Figure 4 FIG. Figure 4 is a schematic flowchart of another resource estimation method provided by an embodiment of the present invention. Referring to Step 103 includes:

[0066] Step 103-1: Through the data scale ladder of the actual measurement use case and the corresponding execution parameter sequence, test and obtain the actual measured peak value of the memory resource occupancy of the actual measurement use case. Among them, the data scale ladder is a set of ordered, stepped data volume levels set to simulate and estimate the operating state of the operator under different data volumes during the estimation process of step 103. By defining multiple increasing or decreasing data volume points, it helps to test and analyze the trend of the memory resource occupancy of the operator changing with the data scale, so as to provide a quantitative basis for performance optimization and resource planning.

[0067] In addition, the execution parameter sequence is used to reflect parameter information such as the service type and time period corresponding to the actual measurement use case.

[0068] Step 103-2: Taking the data scale as the independent variable and the actual measured peak value of the memory resource occupancy as the dependent variable, obtain the corresponding actual measured value of the memory resource occupancy through a linear regression model. Optionally, if the linear regression model is an Ordinary Least Squares (OLS) linear regression model, then taking the data scale as the independent variable and the actual measured peak value of the memory resource occupancy as the dependent variable, obtain the actual measured value of the memory resource occupancy through the ordinary least squares linear regression model.

[0069] Optionally, in order to reduce the interference of JVM fluctuations on the actual measured value of the memory resource occupancy, the actual measured value sequence corresponding to the memory resource occupancy of the minimum data volume of the operator can be generated through multi-batch actual measurement sampling. That is, in some scenarios, the actual measured value of the memory resource occupancy can be multiple values in a sequence.

[0070] Optionally, for step 104, a possible way is:

[0071] Optionally, ​Perform Bayesian fusion on the theoretical value of memory resource occupancy and the corresponding measured value of memory resource occupancy to obtain the posterior distribution.

[0072] Take the mean of the posterior distribution to obtain the predicted value of the memory resource occupancy of the operator.

[0073] Among them, for the above Bayesian fusion, a possible implementation method is provided below. Specifically, the posterior distribution satisfies the following formula:

[0074] Among them, is the posterior distribution that fuses the theoretical value of memory resource occupancy and the measured value of memory resource occupancy.

[0075] represents the true memory resource occupancy value, and this can be understood as a random variable to be inferred.

[0076] is the measured value of memory resource occupancy; optionally, can be the measured value of memory resource occupancy corresponding to the actual running observation data in the above example, such as a sequence of measured values of memory resource occupancy .

[0077] is the probability distribution of the theoretical value of memory resource occupancy. Optionally, this can be understood as a prior distribution.

[0078] is the likelihood function. Optionally, this represents the probability of observing the measured value of memory resource occupancy given the true memory resource occupancy value.

[0079] is the marginal probability. Optionally, this can be a normalization constant to ensure that the sum of posterior probabilities is 1.

[0080] Optionally, a possible example of data modeling is provided below: For , it is a prior distribution: 1. Assume that the theoretical value of memory resource occupancy is the initial estimate of the true memory resource occupancy value , but there is a systematic bias.

[0081] 2. Model it as a normal distribution: ; Among them, (the theoretical value of memory resource occupancy is used as the prior mean); : Robust variance, which reflects the uncertainty of the theoretical estimate (the initial value is set to 5% of the theoretical value of the memory resource occupancy, i.e., ).

[0082] For the likelihood function : 1. Assume that each measurement is independent and identically distributed (i.i.d.) and affected by random noise.

[0083] 2. Model it as a normal distribution: ; where, : is the true memory resource occupancy value (the mean of the likelihood function); : the measurement noise variance (estimated from the actual measured values of the memory resource occupancy, such as the sample standard deviation).

[0084] 3. Joint likelihood (n independent measurements): .

[0085] For the marginal probability : 1. Integrate and normalize : .

[0086] 2. Since and are both normal distributions, is also a normal distribution (the specific form is omitted as it only serves as a normalization constant in Bayes' formula).

[0087] For the derivation of the posterior distribution : 1. Substitute the prior and likelihood of the normal distribution into Bayes' formula, and the posterior distribution is still a normal distribution (conjugate prior property): .

[0088] 1.1. The expression for the posterior mean is:

[0089] where, (the mean of the actual measured values of the memory resource occupancy corresponding to the actual running observation data); the numerator is the weighted sum of the prior mean and the observation mean, and the weights are the reciprocals of their respective variances; the denominator is the normalization coefficient.

[0090] 1.2. The expression for the posterior variance is:

[0091] Among them, the posterior variance reflects the uncertainty after fusion: the smaller the prior variance and the measurement variance, the smaller the posterior variance (the more accurate the estimation).

[0092] Robust variance Function: 1. Initial setting: (Uncertainty of 5% of the theoretical memory resource occupancy); 2. Dynamic adjustment (optional): 2.1. If the measured data fluctuates greatly ( large), the can be appropriately increased. The expression for reducing the prior weight is:

[0093] The change of this prior weight is to prevent the theoretical value of memory resource occupancy from overly dominating the posterior estimation.

[0094] The implementation steps of the corresponding Bayesian fusion algorithm are as follows: A. Initialization: A1. Input the theoretical value of memory resource occupancy , and set the initial

[0095] A2. Initialize an empty observation list

[0096] Among them, this observation list is used to store the actual running observation data of subsequent measured use cases.

[0097] B. Add actual running observation data (measured value of memory resource occupancy): B1. Each time the measured value of memory resource occupancy is added to the list C. Calculate the posterior distribution: C1. Calculate the observation mean and the sample standard deviation

[0098] C2. Calculate and

[0099] C3. Return the posterior distribution parameters, that is, the predicted value of memory resource occupancy (posterior mean + posterior variance) D. Persist metadata: D1. Store as the final predicted value of memory resource occupancy, as the confidence interval width.

[0100] For the implementation process of the above Bayesian fusion algorithm, a calculation example is provided: Hypothesis: The theoretical value of memory resource occupancy M theory = 1000 KB, initial (KB) (5%), the sequence corresponding to the measured value of memory resource occupancy: (KB).

[0101] Then the calculation steps are as follows: 1. The calculation result of the mean value of the measured value of memory resource occupancy (the predicted value of memory resource occupancy): (KB).

[0102] 2. The calculation result of the variance of the measured value of memory resource occupancy: (KB 2 ) Among them, the standard deviation (KB) 3. The calculation result of the posterior mean:

[0103] Among them, the calculation result of the posterior variance:

[0104] Furthermore, the corresponding predicted value of memory resource occupancy is: $1002.1 \pm 8.16$ KB (95% confidence interval).

[0105] For the above Bayesian fusion algorithm, its core lies in balancing the theoretical expectation and the actual measurement through the weighted average of the theoretical value of memory resource occupancy and the measured value of memory resource occupancy. The role of the above robust variance is to control the credibility of the theoretical value of memory resource occupancy and avoid the wrong theoretical value from dominating the estimation result of this application. Optionally, for dynamic adjustment, the prior weight can be adaptively adjusted according to the observation noise.

[0106] The above Bayesian fusion algorithm, compared with simply using the theoretical value of memory resource occupancy, effectively corrects the deviation in the theoretical value of memory resource occupancy caused by unmodeled factors such as JVM optimization and GC overhead (referring to the system resources consumed during the garbage collection process, such as CPU time, memory occupancy, and program response latency) by absorbing actual running observation data (actual test cases). This makes the predicted value of memory resource occupancy more in line with the actual environment performance. Compared with the predicted value of memory resource occupancy obtained once or multiple times, Bayesian fusion can eliminate the influence of measurement noise and instantaneous fluctuations through probability modeling, providing an estimation result that is both close to reality and has statistical stability. The attached confidence interval can also intuitively reflect the prediction reliability, providing a quantitative basis for resource allocation decisions. This fusion mechanism overcomes the limitation of the theoretical model being divorced from reality and avoids the interference of random fluctuations in the original actual running observation data, achieving a balance among accuracy, robustness, and interpretability in engineering practice.

[0107] In the data analysis scenario in the financial field, for the convenience of managing the predicted value of memory resource occupancy, metadata is constituted based on the predicted value of memory resource occupancy. Furthermore, the metadata of the operator can be constituted in the form of a Directed Acyclic Graph (DAG for short). Optionally, the following provides a possible implementation method for persistent storage. Specifically, Figure 2 On the basis of Figure 5 is the schematic flowchart of another resource prediction method provided by the embodiment of the present invention. Refer to Figure 5 , after step 104, it further includes: Step 105: Constituting the operator metadata of each operator from the predicted value of memory resource occupancy of each operator, the minimum data volume for each operator to run, and the basic operator information corresponding to each operator.

[0108] Among them, the basic operator information may include but is not limited to: the identification information of the operator and the descriptive information of the operator, where the identification information of the operator is used to uniquely index the operator.

[0109] Step 106: Determining the operator corresponding to each data analysis directed acyclic graph DAG among all the operator metadata.

[0110] Step 107: Determining the corresponding critical path according to the operator corresponding to each data analysis DAG.

[0111] Among them, the critical path represents the path with the maximum memory resource occupancy in the corresponding data analysis DAG; each path is composed of multiple operators in the data analysis DAG. Specifically, Figure 6A schematic diagram of a data analysis DAG provided by an embodiment of the present invention is shown in Figure 6 which includes multiple vertices (operators) and directed edges. Among them, the operators include: Standard Deviation of Return on Investment (I is a prefix, representing the indicator Indicator, R is the abbreviation of Return, representing the return on investment, STD is the abbreviation of Standard Deviation, hereinafter referred to as IR_STD), Standard Deviation (Standard Deviation, hereinafter referred to as STD), Return on Investment Indicator (Indicator Return, hereinafter referred to as IR), Profit and Loss Indicator (Indicator Profit and Loss, hereinafter referred to as IPL), Profit and Loss Data (Profit and Loss, hereinafter referred to as PL), Return Algorithm (division, DIV), and Cost Data (COST).

[0112] Among them, the following relationships exist between the operators: IR_STD and IR, STD: IR_STD depends on IR and STD, that is, the standard deviation of the return on investment is obtained by performing a standard deviation operation on the return on investment indicator, which can be expressed as IR_STD = STD(IR).

[0113] IR and IPL, DIV, COST: IR depends on IPL, DIV, and COST, that is, the return on investment is obtained by dividing the profit and loss by the cost, IR = IPL / COST.

[0114] IPL and PL: IPL depends on PL, and the calculation of the profit and loss indicator requires obtaining the profit and loss data.

[0115] Correspondingly, the possible paths include: Path 1 (IR_STD → IR → IPL → PL): This path involves multiple complex calculation nodes, especially the PL node, which may need to process a large amount of transaction data and may have a high memory occupancy.

[0116] Path 2 (IR_STD → IR → DIV): This path involves the IR_STD, IR, and DIV nodes and has a medium memory occupancy.

[0117] Path 3 (IR_STD → IR → COST): This path involves the IR_STD, IR, and COST nodes and has a medium memory occupancy.

[0118] Path 4 (IR_STD → STD): This path involves the IR_STD and STD nodes and has a high memory occupancy, especially the calculation of the STD node.

[0119] Step 108: Determine the peak estimated memory resource occupancy of the data analysis DAG corresponding to the data analysis DAG based on all the operators included in the critical path.

[0120] For example, continue to refer to the above Figure 6 , the memory resource occupancy of path 1 is the largest, so this path is the critical path in the above data analysis DAG example. Therefore, based on all the operators included in this path 1: IR_STD, IR, IPL, and PL, determine the peak estimated memory resource occupancy of the data analysis DAG.

[0121] It should be noted that for steps 106 to 108, a mechanism for dynamically programming the peak estimated memory resource occupancy of the data analysis DAG based on the data analysis DAG is formed. The peak estimated memory resource occupancy of the corresponding data analysis DAG can be dynamically estimated based on different data analysis DAGs, thereby further improving the matching degree between the peak memory resource occupancy and the memory resources.

[0122] Furthermore, when determining the target estimated memory resource occupancy value, the peak estimated memory resource occupancy of the data analysis DAG corresponding to the target calculation request can be calculated based on this target, and the peak memory resource occupancy corresponding to the target calculation request can be determined. Specifically, continue to refer to Figure 5 , step 109, includes: Step 109-1: When the target calculation request is obtained, determine the peak memory resource occupancy corresponding to the target calculation request based on the target execution parameter sequence of the target calculation request, the data analysis DAG corresponding to the target calculation request, and the peak estimated memory resource occupancy of the data analysis DAG.

[0123] Optionally, to facilitate the logical management and monitoring of the heap memory resources, the heap memory resources can be divided into an initial area (InitialHeap, abbreviated as: IH), a running area (RuntimeHeap, abbreviated as: RH), an available area (AvailableHeap, abbreviated as: AH), and a fault-tolerance area (Fault-tolerance‌‌Heap, abbreviated as: FH).

[0124] Among them, the initial area is the used heap memory resources after the system starts; the running area is the total estimated resource occupancy of the target calculation request; the available area is the logically available heap memory resources of the system; the fault-tolerance area is the heap memory resources multiplied by the fault-tolerance coefficient, and the fault-tolerance coefficient is a constant ratio used for fault-tolerance estimation errors and GC fragmentation. Optionally, the initial value of this fault-tolerance coefficient is set to 5%.

[0125] In a possible implementation, the above heap memory resources satisfy the relationship: Heap = IH + RH + AH + FH Among them, the capacity of each partition is recorded and updated in real time in the memory, and the real available heap memory is monitored by periodic polling to correct the available area.

[0126] Furthermore, the following provides a task scheduling mechanism based on the above-mentioned heap memory resource partitioning. Specifically, on the Figure 2 basis, Figure 7 This is a schematic flowchart of another resource estimation method provided by an embodiment of the present invention. Refer to Figure 7 , step 110, includes: Step 110-1: Compare the available area with the peak memory resource occupancy.

[0127] Specifically, if the available area is greater than the peak memory resource occupancy, execute step 111; if the available area is less than or equal to the peak memory resource occupancy, execute step 112.

[0128] Step 111: Invoke the computing engine to analyze and calculate the target computing request.

[0129] Step 112: Put the target computing request into the buffer queue to wait.

[0130] Optionally, after step 111-1, after the computing engine completes the analysis and calculation, update the current request in the status cache to completed, update AH = AH + M’, update RH = RH – M’ (when the request ends, it is considered that all the resources estimated for this request are recyclable). Where M’ is the peak memory resource occupancy in step 110-1. Furthermore, the buffer queue polling mode dequeues for resource trial calculation until AH > the next M’ is satisfied, and repeat steps 110-1 and subsequent steps until the computing engine completes the analysis and calculation.

[0131] Optionally, for the computing engine, it is a component or framework responsible for executing computing tasks, used for computing process control, data processing, and algorithm operation. The computing engine is the main location where OOM occurs in the system. Although the patent solution avoids the occurrence of OOM at the limit, it still cannot guarantee 100%. Therefore, a solution for automatic recovery and automatic adjustment of the estimation model needs to be designed.

[0132] Optionally, the following provides an implementation solution for automatic recovery and automatic adjustment of the estimation model: First, calculate the configuration parameters of the computing engine.

[0133] For example, XX:OnOutOfMemoryError="restart_script.sh", this configuration parameter is used to automatically restart when OOM occurs in the system, where "restart_script.sh" is the system restart script, and the script execution will record the OOM restart information at the same time.

[0134] Furthermore, after the OOM system restarts, increase the above fault tolerance coefficient, with an adjustment step of 5% and an adjustment upper limit of 20%. When OOM still occurs after reaching the upper limit, the system rejects similar requests and sends a notice to the operation and maintenance management for intervention and handling.

[0135] Finally, after successfully adjusting the fault tolerance coefficient, increase the above robust variance, with a step size set at 1%, and automatically perform iterative update of the posterior.

[0136] To implement each step and the corresponding technical effects in the above examples, the following provides a possible implementation of a resource estimation device. Specifically, Figure 8 The following is a schematic diagram of a resource estimation device provided by an embodiment of the present invention. Refer to Figure 8 The resource estimation device 200 includes: a resource estimation model 201 and a task manager 202.

[0137] The resource estimation model 201 is used to estimate the theoretical value of the memory resource occupancy of any operator during the analysis and calculation process; the operator corresponds to the minimum calculation unit of data during the analysis and calculation process; according to the measured use case and the corresponding execution parameter sequence, estimate the measured value of the memory resource occupancy of the corresponding operator during the analysis and calculation process; perform Bayesian fusion on the theoretical value of the memory resource occupancy and the corresponding measured value of the memory resource occupancy to obtain the estimated value of the memory resource occupancy of the operator. The task manager 202 is used to, when obtaining a target calculation request, determine the peak value of the memory resource occupancy corresponding to the target calculation request according to the target execution parameter sequence of the target calculation request, at least one target operator corresponding to the target calculation request, and the target estimated values of the memory resource occupancy corresponding to all target operators; determine whether the current memory resource capacity meets the peak value of the memory resource occupancy. If it meets, call the calculation engine to perform analysis and calculation on the target calculation request; if it does not meet, put the target calculation request into the buffer queue to wait.

[0138] Optionally, the resource estimation model 201 is specifically used for: Recursively traverse the abstract syntax tree corresponding to each operator to obtain the recursive level information corresponding to each operator; obtain the theoretical value of the resource occupancy of various objects corresponding to each operator and the number of objects; determine the theoretical value of the memory resource occupancy according to the recursive level information, the theoretical value of the resource occupancy of various objects, and the number of objects.

[0139] Optionally, the resource estimation model 201 is specifically used for: By setting the data scale ladder of the measured use case and the corresponding execution parameter sequence, determine the data scale of the measured use case; input the data scale into the linear regression model to obtain the corresponding measured value of the memory resource occupancy.

[0140] Optionally, the resource estimation model 201 is specifically used for: Perform Bayesian fusion on the theoretical value of memory resource occupancy and the corresponding measured value of memory resource occupancy to obtain the posterior distribution. Take the mean of the posterior distribution to obtain the predicted value of the memory resource occupancy of the operator.

[0141] To facilitate the management of the predicted value of memory resource occupancy, in a possible implementation, metadata is formed based on the predicted value of memory resource occupancy. Specifically, on the basis of Figure 8 Figure 9 FIG. is a schematic diagram of another resource prediction device provided by an embodiment of the present invention. Refer to Figure 9 , the resource prediction device 200 further includes: a metadata manager 203 and a resource manager 204.

[0142] The metadata manager 203 is used to form the operator metadata of each operator by the predicted value of the memory resource occupancy of each operator, the minimum data volume for each operator to run, and the basic operator information corresponding to each operator; in all the operator metadata, determine the operators corresponding to each data analysis directed acyclic graph (DAG); determine the corresponding critical path according to the operators corresponding to each data analysis DAG; the critical path represents the path with the largest memory resource occupancy in the corresponding data analysis DAG; each path is composed of multiple operators in the data analysis DAG; determine the predicted peak value of the memory resource occupancy of the data analysis DAG corresponding to the data analysis DAG according to all the operators included in the critical path.

[0143] Optionally, the task manager 202 is specifically used for: When obtaining a target calculation request, determine the peak value of the memory resource occupancy corresponding to the target calculation request according to the target execution parameter sequence of the target calculation request, the data analysis DAG corresponding to the target calculation request, and the predicted peak value of the memory resource occupancy of the data analysis DAG.

[0144] Optionally, the resource manager 204 is used to divide the heap memory resources into an initial area (IH), a running area (RH), an available area (AH), and a fault-tolerant area (FH); where the initial area is the used heap memory resources after the system starts; the running area is the total predicted value of the resource occupancy of the target calculation request; the available area is the logically available heap memory resources of the system; the fault-tolerant area is the heap memory resources multiplied by a fault-tolerant coefficient, and the fault-tolerant coefficient is a constant ratio.

[0145] Optionally, the resource manager 204 is specifically used for: Compare the available area with the peak value of the memory resource occupancy; if the available area is greater than the peak value of the memory resource occupancy, call the computing engine to perform analysis and calculation on the target calculation request; if the available area is less than or equal to the peak value of the memory resource occupancy, put the target calculation request into the buffer queue to wait.

[0146] Next, to more fully illustrate the resource estimation device shown in the above embodiments of the present application, the specific functions and data flows of each model and module of the resource estimation device are described by way of example. Based on Figure 9 , Figure 10 FIG. Figure 10 shows a schematic diagram of another resource estimation device provided by an embodiment of the present invention. The resource estimation device 200 further includes: a database 205 and a calculation engine 206.

[0147] Specifically, the resource estimation model 201 can adopt the "theoretical calculation method" shown above to obtain the theoretical value of the memory resource occupation of any operator during the analysis and calculation process. And the "actual measurement regression method" shown above can be used to obtain the actual measurement value of the memory resource occupation of the corresponding operator during the analysis and calculation process. Then, through "Bayesian fusion", the theoretical value of the memory resource occupation and the actual measurement value of the memory resource occupation are fused into the estimated value of the memory resource occupation of the operator.

[0148] The metadata manager 203 forms the operator metadata of each operator by combining the estimated value of the memory resource occupation of each operator, the minimum data volume for each operator to run, and the basic operator information corresponding to each operator.

[0149] Furthermore, since the data analysis DAG can cover multiple operators, the metadata manager 203 can convert the operator metadata into the DAG metadata of the corresponding data analysis DAG based on the data form of the data analysis DAG.

[0150] Optionally, the metadata manager 203 can persistently store the operator metadata and the DAG metadata as metadata in the database 205 for subsequent reuse.

[0151] Optionally, the database 205 can also store the actual operation observation data, that is, the actual measurement data shown in the figure, and the metadata corresponding to all objects and their attribute structure information.

[0152] Optionally, when the resource estimation model 201 performs the "dynamic programming" shown above, it can call the corresponding operator metadata from the metadata manager 203 to determine the operators corresponding to each data analysis directed acyclic graph DAG, and then determine the corresponding critical path according to the operators corresponding to each data analysis DAG; according to all the operators included in the critical path, determine the estimated peak value of the memory resource occupation of the data analysis DAG corresponding to the data analysis DAG.

[0153] Optionally, in the "initialization" involved in the above example, the metadata manager 203 can be initialized through the database 205.

[0154] Specifically, this initialization may refer to: reading the metadata in the database 205 into the program memory as a cache to improve the access speed and reduce the direct access to the database 205.

[0155] Optionally, when obtaining the target computing request, the task manager 202 may perform a resource trial calculation, which is step 109 above. Optionally, the target execution parameter sequence involved in step 109, at least one target operator corresponding to the target computing request, and the predicted value of the target memory resource occupancy corresponding to all target operators can all be obtained from the metadata manager 203. Optionally, it may be operator metadata obtained based on the operator granularity or DAG metadata obtained based on the DAG granularity.

[0156] Furthermore, the task manager 202 determines whether the resources are sufficient through step 110 above. If the resources are insufficient, the target computing request is placed in the buffer queue. If the resources are sufficient, the computing engine 206 is called to analyze and calculate the target computing request. And after the calculation is completed, the computing engine 206 updates the status cache.

[0157] The embodiment of the present invention also provides an electronic device, which can execute the steps of all the above examples of the embodiment of the present invention to achieve the corresponding technical effects. Specifically, Figure 11 is a schematic structural diagram of an electronic device provided by an embodiment of the present invention. Refer to Figure 11 The electronic device 30 includes: a memory 301 and a processor 300; The memory 301 is used to store one or more programs; The processor 3300; When one or more programs are executed by the processor, when the electronic device 30 is used to execute the steps shown in the above various method examples, it can achieve each step and the corresponding technical effects.

[0158] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architectures, functions, and operations of devices, methods, and computer program products according to multiple embodiments of the present application. In this regard, each block in the flowchart or block diagram may represent a module, a program segment, or a part of code, and the module, program segment, or part of code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than that marked in the accompanying drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and / or flowchart, as well as the combination of blocks in the block diagram and / or flowchart, can be implemented by a dedicated hardware-based system for performing the specified functions or actions, or can be implemented by a combination of dedicated hardware and computer instructions.

[0159] In addition, in each embodiment of the present application, the functional modules can be integrated together to form an independent part, or each module can exist alone, or two or more modules can be integrated to form an independent part.

[0160] If the function is implemented in the form of a software functional module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art, or a part of this technical solution, can be embodied in the form of a program product. This program product is stored in a computer-readable storage medium and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods in each embodiment of the present application. The aforementioned storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), magnetic disks, or optical discs that can store program codes.

[0161] The solution provided in the above example of the present invention can, compared with using Spark or Flink as a heavyweight computing engine in the prior art, more lightweight, flexible, and accurately estimate the resources corresponding to the computing tasks, and then quantitatively complete the scheduling of the computing tasks. This solves the problem in the prior art solution that it is impossible to estimate the resource requirements of computing requests, or the deviation between the estimation and the actual resources and resource requirements is relatively large.

[0162] At the same time, even if OOM occurs, the estimation mechanism of this solution can be iterated by increasing the above-mentioned fault tolerance coefficient and robust variance without the need for human intervention, tuning, and introduction of heavyweight middleware, thereby further reducing the OOM risk and the overall cost of using the solution.

[0163] In the stress load test and capacity test verification, by monitoring the JVM memory usage data and comparing the estimated data, the root mean square error (RMSE) is around 3%-7%, which is at a relatively low error level. In actual production applications, this solution effectively ensures system reliability and reduces the probability of OOM.

[0164] In this patent solution, the adjustment direction and amplitude of the Bayesian fusion result depend on the relative credibility of the two: if the theoretical value variance (robust variance) is small, the fusion result is closer to the theoretical value; if the measured value variance is small (measurement is accurate), it is more inclined to the measured value. This adaptive weighting makes Bayesian fusion suitable for handling the quantification problem of uncertainty in memory resource usage, and can be used to design a more extensive memory resource management quantification solution.

[0165] The above are only preferred embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

[0166] It will be apparent to those skilled in the art that the present application is not limited to the details of the exemplary embodiments described above, and that the present application can be implemented in other specific forms without departing from the spirit or essential features of the present application. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the present application is defined by the appended claims rather than the above description, and it is intended that all changes falling within the meaning and scope of the equivalent elements of the claims be included in the present application. Any reference numeral in a claim should not be considered as limiting the claim to which it relates.

Claims

1. A resource estimation method, characterized in that, Including: Estimating the theoretical value of the memory resource occupation of any operator during the analysis and calculation process; The operator corresponds to the smallest calculation unit for data in the analysis and calculation process; According to the measured use case and the corresponding execution parameter sequence, estimating the measured value of the memory resource occupation of the corresponding operator during the analysis and calculation process; Performing Bayesian fusion on the theoretical value of the memory resource occupation and the corresponding measured value of the memory resource occupation to obtain the estimated value of the memory resource occupation of the operator; When a target calculation request is obtained, determining the peak value of the memory resource occupation corresponding to the target calculation request according to the target execution parameter sequence of the target calculation request, at least one target operator corresponding to the target calculation request, and the estimated values of the target memory resource occupation corresponding to all the target operators; Determining whether the current memory resource capacity meets the peak value of the memory resource occupation. If it meets, calling the calculation engine to perform analysis and calculation on the target calculation request; If it does not meet, putting the target calculation request into the buffer queue to wait.

2. The method according to claim 1, wherein The step of estimating the theoretical value of the memory resource occupation of any operator during the analysis and calculation process includes: Recursively traversing the abstract syntax tree corresponding to each operator to obtain the recursive level information corresponding to each operator; Obtaining the theoretical value of the resource occupation and the number of objects of various types corresponding to each operator; Determining the theoretical value of the memory resource occupation according to the recursive level information, the theoretical value of the resource occupation of various types of objects, and the number of objects.

3. The method according to claim 1, characterized in that, The step of estimating the measured value of the memory resource occupation of the corresponding operator during the analysis and calculation process according to the measured use case and the corresponding execution parameter sequence includes: Testing to obtain the measured peak value of the memory resource occupation of the measured use case through the data scale ladder of the measured use case and the corresponding execution parameter sequence; Taking the data scale as the independent variable and the measured peak value of the memory resource occupation as the dependent variable, and obtaining the corresponding measured value of the memory resource occupation through a linear regression model.

4. The method according to claim 1, wherein The step of performing Bayesian fusion on the theoretical value of the memory resource occupation and the corresponding measured value of the memory resource occupation to obtain the estimated value of the memory resource occupation of the operator includes: Performing Bayesian fusion on the theoretical value of the memory resource occupation and the corresponding measured value of the memory resource occupation to obtain the posterior distribution; Taking the mean of the posterior distribution to obtain the estimated value of the memory resource occupation of the operator.

5. The method according to claim 1, characterized in that, After the step of performing Bayesian fusion on the theoretical value of the memory resource occupation and the corresponding measured value of the memory resource occupation to obtain the estimated value of the memory resource occupation of the operator, it further includes: Constructing the operator metadata of each operator by the estimated value of the memory resource occupation of each operator, the minimum data volume for each operator to run, and the basic operator information corresponding to each operator; Determining the operators corresponding to each data analysis directed acyclic graph (DAG) among all the operator metadata; Determining the corresponding critical path according to the operators corresponding to each data analysis DAG; the critical path represents the path with the largest memory resource occupation in the corresponding data analysis DAG; each path is composed of multiple operators in the data analysis DAG; Determine the peak estimated memory resource occupancy of the data analysis DAG corresponding to the data analysis according to all the operators included in the critical path.

6. The method according to claim 5, wherein When a target calculation request is obtained, the steps of determining the peak memory resource occupancy corresponding to the target calculation request according to the target execution parameter sequence of the target calculation request, at least one target operator corresponding to the target calculation request, and the target memory resource occupancy estimated values corresponding to all the target operators include: When a target calculation request is obtained, determine the peak memory resource occupancy corresponding to the target calculation request according to the target execution parameter sequence of the target calculation request, the data analysis DAG corresponding to the target calculation request, and the peak estimated memory resource occupancy of the data analysis DAG.

7. The method according to claim 1, wherein It further includes: Divide the heap memory resources into an initial area, a running area, an available area, and a fault-tolerant area; wherein, the initial area is the used heap memory resources after the system starts. The running area is the total estimated resource occupancy of the target calculation request; the available area is the logically available heap memory resources of the system; the fault-tolerant area is the heap memory resources multiplied by a fault-tolerant coefficient, and the fault-tolerant coefficient is a constant ratio.

8. The method according to claim 7, characterized in that, The steps of determining whether the current memory resource capacity meets the peak memory resource occupancy include: Compare the available area with the peak memory resource occupancy. If the available area is greater than the peak memory resource occupancy, call the computing engine to perform analysis and calculation on the target calculation request. If the available area is less than or equal to the peak memory resource occupancy, put the target calculation request into the buffer queue to wait.

9. A resource estimation device, characterized in that, It includes: A resource estimation model and a task manager; The resource estimation model is used to estimate the theoretical value of the memory resource occupancy of any operator during the analysis and calculation process. The operator corresponds to the minimum calculation unit for data during the analysis and calculation process; according to the measured use case and the corresponding execution parameter sequence, estimate the measured value of the memory resource occupancy of the corresponding operator during the analysis and calculation process. Perform Bayesian fusion on the theoretical value of the memory resource occupancy and the corresponding measured value of the memory resource occupancy to obtain the estimated value of the memory resource occupancy of the operator. The task manager is used to, when a target calculation request is obtained, determine the peak memory resource occupancy corresponding to the target calculation request according to the target execution parameter sequence of the target calculation request, at least one target operator corresponding to the target calculation request, and the target memory resource occupancy estimated values corresponding to all the target operators. Determine whether the current memory resource capacity meets the peak memory resource occupancy. If it meets, call the computing engine to perform analysis and calculation on the target calculation request; if it does not meet, put the target calculation request into the buffer queue to wait.

10. The device according to claim 9, characterized in that, It further includes: A metadata manager; The metadata manager is used to form the operator metadata of each operator by combining the estimated value of the memory resource occupancy of each operator, the minimum data volume for each operator to run, and the basic operator information corresponding to each operator. Among all the operator metadata, determine the operators corresponding to each data analysis directed acyclic graph (DAG). Determine the corresponding critical path according to the operators corresponding to each of the data analysis DAGs; the critical path represents the path with the largest memory resource occupancy in the corresponding data analysis DAG; each of the paths is composed of multiple operators in the data analysis DAG; determine the estimated peak value of the memory resource occupancy of the data analysis DAG corresponding to the data analysis DAG according to all the operators included in the critical path.

11. The device according to claim 9, characterized in that, Further comprising: A resource manager; The resource manager is used to divide the heap memory resources into an initial area, a running area, an available area, and a fault-tolerant area; wherein, the initial area is the heap memory resources already used after the system starts; the running area is the total estimated resource occupancy of the target calculation request; the available area is the logically available heap memory resources of the system; the fault-tolerant area is the heap memory resources multiplied by a fault-tolerant coefficient, and the fault-tolerant coefficient is a constant ratio.

12. An electronic device, characterized in that, Comprising: A memory for storing one or more programs; A processor; When the one or more programs are executed by the processor, the method according to any one of claims 1-8 is implemented.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, the method according to any one of claims 1-8 is implemented.

14. A program product, characterized in that, When the program product is executed by the processor, the method according to any one of claims 1-8 is implemented.

Images