A resource estimation method and program product

Abstract

The present application proposes a resource estimation method and program product, in which the memory resource occupancy estimation value is defined as the optimal estimate of the memory resource occupancy theoretical value and the memory resource occupancy measured value through Bayesian fusion. The memory resource occupancy estimation value not only includes the best predicted value of the memory resource occupancy, but also fully retains the uncertainty information of the estimate. Compared with simply using the theoretical value of memory resource occupancy, the result of this Bayesian fusion effectively corrects the deviation caused by unmodeled factors in the theoretical value of memory resource occupancy by absorbing actual operation observation data, making the memory resource occupancy estimation value more consistent with the actual environment performance. This provides a memory resource occupancy estimation value that is both close to reality and statistically stable, and ultimately realizes the estimation of resources corresponding to computing tasks in a more lightweight, flexible and accurate manner, thereby completing the scheduling of computing tasks in a quantifiable manner.

Description

Technical Field

[0001] The present application relates to the field of data computing, and in particular to a resource estimation method and program product. Background Art

[0002] The financial sector faces a massive demand for data analysis, and the analytical computing scenarios are complex and dynamic. For example, the analysis objects, dimensions, and calculation calibers all experience dynamic changes. Furthermore, as business data grows, the scale and concurrency of analytical computing also become random. Data analysis systems handle both high-concurrency, low-latency computing and large-scale, high-latency computing, placing higher demands on the system's memory management.

[0003] In existing technologies, the risk of out of memory (OOM) during computing is usually reduced through program optimization, parameter tuning, and request rate limiting.

[0004] However, the optimization mechanisms of existing technologies often adopt fixed configurations based on experience, which have poor adaptability to dynamic scenarios and random sudden changes and often require human intervention, thereby reducing efficiency and resource utilization. Summary of the Invention

[0005] The purpose of this application is to provide a resource estimation method and program product, which is used to provide a memory resource occupancy estimate that is both close to reality and statistically stable, thereby quantifying the scheduling of computing tasks.

[0006] In order to achieve the above objectives, the technical solutions adopted in the embodiments of the present application are as follows:

[0007] In a first aspect, an embodiment of the present application provides a resource estimation method, comprising:

[0008] Estimate the theoretical value of memory resource usage of any operator during the analysis and calculation process; the operator corresponds to the minimum calculation unit for data in the analysis and calculation process;

[0009] Based on the measured use cases and the corresponding execution parameter sequences, estimate the measured value of memory resource usage of the corresponding operator during the analysis and calculation process;

[0010] Performing Bayesian fusion on the theoretical value of memory resource occupancy and the corresponding measured value of memory resource occupancy to obtain an estimated value of memory resource occupancy of the operator;

[0011] When a target computing request is obtained, determining a peak memory resource occupancy value corresponding to the target computing request based on a target execution parameter sequence of the target computing request, at least one target operator corresponding to the target computing request, and target memory resource occupancy estimates corresponding to all the target operators;

[0012] Determine whether the current memory resource capacity meets the memory resource occupancy peak, and if so, call the computing engine to analyze and calculate the target computing request;

[0013] If not, the target computing request is put into a buffer queue and waits.

[0014] Optionally, the step of estimating a theoretical value of memory resource usage of any operator during the analysis and calculation process includes:

[0015] Recursively traverse the abstract syntax tree corresponding to each operator to obtain the recursive level information corresponding to each operator;

[0016] Obtaining theoretical resource occupancy values ​​and object quantities of various objects corresponding to each operator;

[0017] The theoretical value of memory resource occupation is determined according to the recursive level information, the theoretical value of resource occupation of each type of object, and the number of objects.

[0018] Optionally, the step of estimating the measured value of memory resource usage of the corresponding operator during the analysis and calculation process based on the measured use case and the corresponding execution parameter sequence includes:

[0019] By testing the data scale ladder and the corresponding execution parameter sequence of the measured use case, the measured peak value of the memory resource usage of the measured use case is obtained;

[0020] The data scale is used as the independent variable, and the measured peak value of the memory resource occupancy is used as the dependent variable, and the corresponding measured value of the memory resource occupancy is obtained through a linear regression model.

[0021] Optionally, the step of performing Bayesian fusion on the theoretical value of memory resource occupancy and the corresponding measured value of memory resource occupancy to obtain an estimated value of memory resource occupancy of the operator includes:

[0022] Performing Bayesian fusion on the theoretical value of memory resource occupancy and the corresponding measured value of memory resource occupancy to obtain a posterior distribution;

[0023] The posterior distribution is averaged to obtain an estimated memory resource occupancy value of the operator.

[0024] Optionally, after the step of performing Bayesian fusion on the theoretical value of memory resource occupancy and the corresponding measured value of memory resource occupancy to obtain an estimated value of memory resource occupancy of the operator, the method further includes:

[0025] The estimated memory resource usage of each operator, the minimum amount of data required for each operator to run, and the basic operator information corresponding to each operator are used to form the operator metadata of each operator;

[0026] Determine the operator corresponding to each data analysis directed acyclic graph (DAG) in all the operator metadata;

[0027] Determine a corresponding critical path based on the operator corresponding to each data analysis DAG; the critical path represents the path with the largest memory resource usage in the data analysis DAG; each path is composed of multiple operators in the data analysis DAG;

[0028] Determine an estimated peak value of data analysis DAG memory resource usage corresponding to the data analysis DAG based on all operators included in the critical path.

[0029] Optionally, when a target computing request is obtained, the step of determining a peak memory resource occupancy corresponding to the target computing request based on a target execution parameter sequence of the target computing request, at least one target operator corresponding to the target computing request, and target memory resource occupancy estimates corresponding to all the target operators includes:

[0030] When a target computing request is obtained, the peak memory resource occupancy corresponding to the target computing request is determined based on the target execution parameter sequence of the target computing request, the data analysis DAG corresponding to the target computing request, and the estimated peak memory resource occupancy of the data analysis DAG.

[0031] Optionally, it also includes:

[0032] The heap memory resources are divided into an initial area, a running area, an available area, and a fault-tolerant area; wherein the initial area is the heap memory resources used after the system is started; the running area is the estimated total resource occupancy of the target computing request; the available area is the heap memory resources available for system logic; and the fault-tolerant area is the heap memory resources multiplied by a fault-tolerant coefficient, where the fault-tolerant coefficient is a constant ratio.

[0033] Optionally, the step of determining whether the current memory resource capacity meets the peak memory resource occupancy includes:

[0034] Comparing the available zone with the peak memory resource usage;

[0035] If the available area is larger than the peak memory resource usage, calling the computing engine to analyze and calculate the target computing request;

[0036] If the available area is less than or equal to the peak memory resource occupancy, the target computing request is placed in a buffer queue for waiting.

[0037] In a second aspect, an embodiment of the present application provides a resource estimation device, comprising: a resource estimation model and a task manager;

[0038] The resource estimation model is used to estimate the theoretical value of memory resource occupancy of any operator during the analysis and calculation process; the operator corresponds to the minimum computing unit for data in the analysis and calculation process; based on the actual measured use case and the corresponding execution parameter sequence, the actual measured value of memory resource occupancy of the corresponding operator during the analysis and calculation process is estimated; the theoretical value of memory resource occupancy and the corresponding actual measured value of memory resource occupancy are Bayesian fused to obtain the estimated value of memory resource occupancy of the operator;

[0039] The task manager is used to determine, when obtaining a target computing request, the peak memory resource occupancy corresponding to the target computing request based on the target execution parameter sequence of the target computing request, at least one target operator corresponding to the target computing request, and the estimated target memory resource occupancy corresponding to all the target operators; determine whether the current memory resource capacity meets the peak memory resource occupancy; if so, call the computing engine to analyze and calculate the target computing request; if not, put the target computing request into a buffer queue for waiting.

[0040] Optionally, the resource estimation model is specifically used to:

[0041] Recursively traverse the abstract syntax tree corresponding to each operator to obtain the recursive level information corresponding to each operator; obtain the theoretical resource occupancy value and the number of objects of each type of object corresponding to each operator; determine the theoretical memory resource occupancy value based on the recursive level information, the theoretical resource occupancy value of each type of object and the number of objects.

[0042] Optionally, the resource estimation model is specifically used to:

[0043] Through the data scale ladder and corresponding execution parameter sequence of the measured use case, the test obtains the measured peak value of memory resource occupancy of the measured use case; taking the data scale as the independent variable and the measured peak value of memory resource occupancy as the dependent variable, the corresponding measured value of memory resource occupancy is obtained through a linear regression model.

[0044] Optionally, it further includes: a metadata manager;

[0045] The metadata manager is used to form the operator metadata of each operator by combining the estimated memory resource occupancy of each operator, the minimum amount of data required for the operation of each operator, and the basic operator information corresponding to each operator; determine the operator corresponding to each data analysis directed acyclic graph (DAG) in all the operator metadata; determine the corresponding critical path according to the operator corresponding to each data analysis DAG; the critical path represents the path with the maximum memory resource occupancy in the data analysis DAG; each path is composed of multiple operators in the data analysis DAG; and determine the estimated peak value of the data analysis DAG memory resource occupancy corresponding to the data analysis DAG based on all the operators included in the critical path.

[0046] Optionally, the task manager is specifically configured to:

[0047] When a target computing request is obtained, the peak memory resource occupancy corresponding to the target computing request is determined based on the target execution parameter sequence of the target computing request, the data analysis DAG corresponding to the target computing request, and the estimated peak memory resource occupancy of the data analysis DAG.

[0048] Optionally, it further includes: a resource manager;

[0049] The resource manager is used to divide heap memory resources into an initial area, a running area, an available area, and a fault-tolerant area; wherein the initial area is the heap memory resources used after the system is started; the running area is the estimated total resource occupancy of the target computing request; the available area is the heap memory resources available for system logic; and the fault-tolerant area is the heap memory resources multiplied by a fault-tolerant coefficient, where the fault-tolerant coefficient is a constant ratio.

[0050] Optionally, the resource manager is specifically configured to:

[0051] Compare the available area with the peak memory resource occupancy; if the available area is larger than the peak memory resource occupancy, call the computing engine to analyze and calculate the target computing request; if the available area is less than or equal to the peak memory resource occupancy, put the target computing request into a buffer queue for waiting.

[0052] In a third aspect, an embodiment of the present application provides an electronic device, including:

[0053] a memory for storing one or more programs;

[0054] processor;

[0055] When the one or more programs are executed by the processor, the method as described in any one of the above-mentioned first aspects is implemented.

[0056] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, characterized in that when the computer program is executed by a processor, it implements the method as described in any one of the above-mentioned first aspects.

[0057] In a fifth aspect, an embodiment of the present application provides a program product, which, when executed by a processor, implements the method as described in any one of the above-mentioned first aspects.

[0058] Compared with the prior art, the embodiment of the present application provides a resource estimation method and program product, in which the memory resource occupancy estimation value is defined as the optimal estimate of the memory resource occupancy theoretical value and the memory resource occupancy measured value through Bayesian fusion. The memory resource occupancy estimation value not only includes the best predicted value of the memory resource occupancy, but also fully retains the uncertainty information of the estimate. Compared with simply using the theoretical value of memory resource occupancy, the result of this Bayesian fusion effectively corrects the deviation caused by unmodeled factors in the theoretical value of memory resource occupancy by absorbing actual operation observation data (actual test case), making the memory resource occupancy estimation value more consistent with the actual environment performance. This provides a memory resource occupancy estimation value that is both close to reality and statistically stable, and ultimately achieves a lighter, more flexible and accurate estimation of the resources corresponding to the computing task, thereby completing the scheduling of the computing task in a quantifiable manner.

[0059] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments. It should be understood that the following drawings only illustrate certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without creative work.

[0061] Figure 1 A schematic diagram of the OOM risk control system in the prior art;

[0062] Figure 2 A schematic diagram of a process flow of a resource estimation method provided by an embodiment of the present invention;

[0063] Figure 3 A flowchart of another resource estimation method provided by an embodiment of the present invention;

[0064] Figure 4 A flowchart of another resource estimation method provided by an embodiment of the present invention;

[0065] Figure 5A flowchart of another resource estimation method provided by an embodiment of the present invention;

[0066] Figure 6 A schematic diagram of a data analysis DAG provided by an embodiment of the present invention;

[0067] Figure 7 A flowchart of another resource estimation method provided by an embodiment of the present invention;

[0068] Figure 8 A schematic diagram of a resource estimation device provided by an embodiment of the present invention;

[0069] Figure 9 A schematic diagram of another resource estimation device provided by an embodiment of the present invention;

[0070] Figure 10 A schematic diagram of another resource estimation device provided by an embodiment of the present invention;

[0071] Figure 11 A schematic structural diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0072] To make the objectives, technical solutions, and advantages of the embodiments of the present application more clear, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Generally, the components of the embodiments of the present application described and shown in the drawings herein can be arranged and designed in various different configurations.

[0073] Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the present application for protection, but merely represents selected embodiments of the present application. All other embodiments obtained by persons of ordinary skill in the art based on the embodiments in the present application without creative work are within the scope of protection of the present application.

[0074] It should be noted that similar reference numerals and letters represent similar items in the following drawings. Therefore, once an item is defined in one drawing, it does not need to be further defined or explained in subsequent drawings. At the same time, in the description of this application, the terms "first", "second", etc. are only used to distinguish the description and should not be understood as indicating or implying relative importance.

[0075] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply the existence of any such actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or device comprising the element.

[0076] In the existing technology, in order to reduce the OOM risk in the data analysis process, it is usually divided into two stages, namely, the risk control mechanism is deployed "before running" and "at running" during the data analysis calculation process. Specifically, Figure 1 This is a schematic diagram of the OOM risk control system in the prior art, see Figure 1 Before running, Java Virtual Machine (JVM) optimization and static parameter tuning are typically used. "Static JVM tuning" refers to optimizing and adjusting the JVM's static parameters to reduce the risk of memory-related issues such as memory leaks during JVM operation, thereby improving JVM stability and performance.

[0077] Furthermore, a request flow limiting scheme is set at "runtime", which can be implemented by setting a corresponding threshold for the upper limit of memory resource usage. Figure 1 As shown, when a client initiates a computation request, the "request interceptor" checks whether the current memory resource usage exceeds a threshold. If the usage exceeds the threshold, indicating insufficient resources, the computation request is cached in a waiting queue. The business processor then executes the corresponding data analysis when the memory resource usage falls below the threshold.

[0078] If the usage rate is less than the threshold, it means that “resources are sufficient”, and the data analysis corresponding to the computing request is performed by the business processor.

[0079] In this solution, the resource manager implements the flow control of computing requests by setting thresholds for memory resources and monitoring resources.

[0080] However, the above method has the following defects:

[0081] 1. Static technical solutions such as parameter tuning are based on conventional business scenarios and solidify a set of configurations based on empirical values. They cannot adapt to various dynamic scenarios and random sudden changes. Manual intervention and adjustment are required after OOM occurs.

[0082] Second, the request rate limiting solution requires setting a monitoring threshold for memory resources. It cannot accurately estimate and distinguish between high-resource requests and low-resource requests, resulting in reduced resource utilization and mistakenly placing some low-resource computing requests into the blocking queue. In addition, large-scale data computing requests may directly break through the threshold and cause OOM.

[0083] Third, as heavyweight computing engines, Spark or Flink require significant additional hardware and operation and maintenance costs, resulting in low feasibility. Similarly, because resource consumption cannot be estimated during the computing process, algorithm complexity explosion, Spark partition data skew, and Flink backpressure failure data accumulation can also directly cause OOM.

[0084] To address the aforementioned technical issues, this application provides a resource estimation mechanism that uses theoretical values ​​representing theoretically expected memory resource usage and measured values ​​representing actual measured data to dynamically estimate the resource usage of computing requests. Based on this estimate, computing tasks for data analysis can be scheduled. This solves existing issues such as the inability to estimate or large estimation deviations, the need for manual intervention and tuning when out-of-memory (OOM) occurs, and the heavyweight introduction of middleware.

[0085] Optionally, the core process of this application is exemplified below. Specifically, Figure 2 A schematic diagram of a resource estimation method according to an embodiment of the present invention is provided. Figure 2 , the method comprising:

[0086] Step 102: Estimate the theoretical value of memory resource usage of any operator during the analysis and calculation process.

[0087] Among them, the operator corresponds to the smallest calculation unit for data in the analysis and calculation process.

[0088] Step 103: Estimate the measured value of memory resource usage of the corresponding operator during the analysis and calculation process based on the measured use case and the corresponding execution parameter sequence.

[0089] The measured use case may include one or more batches of actual operation observation data.

[0090] Step 104: Perform Bayesian fusion on the theoretical value of memory resource usage and the corresponding measured value of memory resource usage to obtain an estimated value of the operator's memory resource usage.

[0091] Among them, Bayesian fusion can combine prior knowledge (such as theoretical memory resource usage values) with new observation data (such as actual memory resource usage values) to obtain results that are closer to the actual situation after analysis, namely, estimated memory resource usage values.

[0092] Step 109: When a target computing request is obtained, the peak memory resource occupancy corresponding to the target computing request is determined based on the target execution parameter sequence of the target computing request, at least one target operator corresponding to the target computing request, and the estimated target memory resource occupancy corresponding to all target operators.

[0093] The target operator is one or more of all operators obtained in step 104 .

[0094] Step 110: Determine whether the current memory resource capacity meets the peak memory resource usage.

[0095] Specifically, if the conditions are met, step 111 is executed; if not, step 112 is executed.

[0096] Step 111: Call the computing engine to analyze and compute the target computing request.

[0097] Step 112: Put the target computing request into a buffer queue and wait.

[0098] Optionally, step 112 may return to step 110 . When the current memory resource capacity is updated, step 110 may be re-judged until the analysis and calculation are completed through step 111 when the conditions are met.

[0099] In the resource estimation method provided in the embodiment of the present application, the memory resource occupancy estimate value is defined as the optimal estimate of the memory resource occupancy theoretical value and the memory resource occupancy measured value through Bayesian fusion. The memory resource occupancy estimate value not only includes the best predicted value of the memory resource occupancy, but also completely retains the uncertainty information of the estimate. Compared with simply using the theoretical value of memory resource occupancy, the result of this Bayesian fusion effectively corrects the deviation caused by unmodeled factors in the theoretical value of memory resource occupancy by absorbing actual operation observation data (actual test case), making the memory resource occupancy estimate value more consistent with the actual environment performance. This provides a memory resource occupancy estimate value that is both close to reality and statistically stable, and ultimately achieves a lighter, more flexible and accurate estimation of the resources corresponding to the computing task, thereby completing the scheduling of the computing task in a quantifiable manner.

[0100] In addition, since the resource estimation method provided in the embodiment of the present application can be embedded and integrated into the application system, thereby achieving integrated deployment with the application system, no additional hardware and operation and maintenance costs are required.

[0101] In one possible implementation, a possible implementation of a "theoretical calculation method" is provided to estimate the theoretical value of memory resource usage of each operator during the analysis and calculation process. Optionally, the "theoretical calculation method" uses an "automatic parsing framework" to recursively traverse the code involved in the operator to obtain all objects and structural information at all levels of the code corresponding to the operator, and then estimates the resource usage of all objects to obtain the theoretical value of memory resource usage of the operator. Specifically, in Figure 2 On the basis of Figure 3 A flow chart of another resource estimation method provided by an embodiment of the present invention is shown in FIG. Figure 3 , step 102, comprising:

[0102] Step 102 - 1 : recursively traverse the abstract syntax tree corresponding to each operator to obtain the recursive level information corresponding to each operator.

[0103] For each operator, its corresponding code can be represented by an abstract syntax tree, which can contain objects at different levels. The size of all objects contained in an operator determines its theoretical memory resource usage. Therefore, by recursively traversing the recursive hierarchy corresponding to the operator, all objects at each level and their attribute structure can be obtained.

[0104] Optionally, for the recursive traversal mechanism, its termination condition may be: when all calls to functions and methods at each level under the abstract syntax tree have been completed, the recursive traversal is terminated.

[0105] Optionally, all objects and their attribute structure information may also be persistently recorded.

[0106] Step 102-2: Obtain the theoretical resource usage value and number of objects of each type corresponding to each operator.

[0107] The theoretical value of the resource usage of the object represents the object size. The object size can be expressed as:

[0108] Single object size = object header + instance data + alignment padding

[0109] Specifically, the object header size depends on the JVM implementation and whether compressed pointers are enabled (usually 12 or 16 bytes); the instance data size is the sum of the space actually occupied by all fields; the alignment padding size = (8 - (object header size + instance data size) % 8) % 8.

[0110] For example, assume a 64-bit JVM with compressed pointers enabled (object header = 12 bytes).

[0111] calculate:

[0112] Object header = 12 bytes

[0113] Instance data = 8 (id) + 4 (name) + 8 (price) = 20 bytes

[0114] Alignment padding = (8 - (12 + 20) % 8) % 8 = (8 - 32 % 8) % 8 = (8 - 0) % 8 = 0 bytes (because 32 is already a multiple of 8).

[0115] Here is a Java example of an object:

[0116] public class Product {

[0117] long id; / / 8 bytes

[0118] string name; / / 4 bytes (quote)

[0119] double price; / / 8 bytes

[0120] }

[0121] Among them, long id, string name, and double price can be the attribute structure information of the object. Obviously, the attribute structure information of the object is different for different examples and is not limited here.

[0122] Step 102 - 3 : Determine the theoretical value of memory resource usage based on the recursive level information, the theoretical value of resource usage of each type of object, and the number of objects.

[0123] Alternatively, most data structures can be object collections or object arrays. The data size and memory resource usage show an approximately linear relationship. The maximum value (process peak) of each recursive level is taken. Therefore, the expression for the theoretical value of memory resource usage can be:

[0124] Mtheory = Max(recursive level information∑(single object size * number of objects))

[0125] In summary, this application uses a theoretical calculation method to estimate the theoretical value of memory resource usage of any operator during the analysis and calculation process. Optionally, the theoretical value of memory resource usage is estimated based on the minimum amount of data when the corresponding operator is running.

[0126] In one possible implementation, in order to better estimate the actual value of memory resource usage of each operator during the analysis and calculation process, the "measured regression method" can be used to achieve specific estimation. Specifically, Figure 2 On the basis of Figure 4 A flow chart of another resource estimation method provided by an embodiment of the present invention is shown in FIG. Figure 4 , step 103, comprising:

[0127] Step 103 - 1 : Through the data scale ladder of the measured use case and the corresponding execution parameter sequence, the measured peak value of the memory resource usage of the measured use case is obtained through testing.

[0128] Among them, the data scale ladder is used in the estimation process of step 103 to set a set of orderly, stepped data volume levels in order to simulate and estimate the operating status of the operator under different data volumes. By defining multiple increasing or decreasing data volume points, it helps to test and analyze the trend of the operator's memory resource usage changing with the data scale, thereby providing a quantitative basis for performance optimization and resource planning.

[0129] In addition, the execution parameter sequence is used to reflect parameter information such as the business type and time period corresponding to the measured use case.

[0130] Step 103 - 2 : Using the data size as the independent variable and the measured peak value of memory resource usage as the dependent variable, obtain the corresponding measured value of memory resource usage through a linear regression model.

[0131] Optionally, if the linear regression model is an Ordinary Least Squares (OLS) linear regression model, the data size is used as the independent variable and the measured peak value of memory resource usage is used as the dependent variable, and the measured value of memory resource usage is obtained through the least squares linear regression model.

[0132] Optionally, to reduce the impact of JVM fluctuations on the measured memory resource usage, multiple batches of measured samples can be used to generate a sequence corresponding to the measured memory resource usage for the operator with the minimum amount of data. In other words, in some scenarios, the measured memory resource usage value can be multiple values ​​in a set of sequences.

[0133] Optionally, for step 104, a possible method is:

[0134] The theoretical value of memory resource usage and the corresponding measured value of memory resource usage are fused by Bayesian method to obtain the posterior distribution.

[0135] Take the mean of the posterior distribution to obtain the estimated memory resource usage of the operator.

[0136] Among them, for the above Bayesian fusion, a possible implementation method is provided below. Specifically, the posterior distribution satisfies the following formula:

[0137]

[0138] in, It is the posterior distribution that integrates the theoretical value of memory resource usage and the measured value of memory resource usage.

[0139] Represents the actual memory resource usage value. can be understood as a random variable to be inferred.

[0140] is the measured value of memory resource usage; optionally, The measured values ​​of memory resource usage corresponding to the actual running observation data in the above example can be, for example, a sequence of measured values ​​of memory resource usage .

[0141] is the probability distribution of the theoretical value of memory resource occupancy. Optionally, the can be understood as a prior distribution.

[0142] is the likelihood function, optionally, the Indicates the probability of observing the measured value of memory resource usage when the actual memory resource usage value is given.

[0143] is the marginal probability, optionally, the It can be a normalization constant to ensure that the posterior probabilities sum to 1.

[0144] Alternatively, here is an example of possible data modeling:

[0145] for , which serves as a prior distribution:

[0146] 1. Assume the theoretical value of memory resource usage Is the actual memory resource usage value However, there are systematic biases in the initial estimate.

[0147] 2. Modeling as a normal distribution: ;

[0148] in, (The theoretical value of memory resource usage is used as the prior mean); : Robust variance, reflecting the uncertainty of theoretical estimates (the initial value is set to 5% of the theoretical value of memory resource occupancy, i.e. ).

[0149] For the likelihood function :

[0150] 1. Assume that each measurement are independent and identically distributed (iid) and are subject to random noise.

[0151] 2. Modeling as a normal distribution: ;

[0152] in, : is the actual memory resource usage value (mean value of likelihood function); : Measurement noise variance (estimated from data on measured values ​​of memory resource usage, such as sample standard deviation).

[0153] 3. Joint likelihood (n independent measurements): .

[0154] For marginal probabilities :

[0155] 1. Yes Integral normalization: .

[0156] 2. Due to and are all normally distributed, It is also a normal distribution (the specific form is omitted because it is only used as a normalization constant in the Bayesian formula).

[0157] For the posterior distribution Derivation of:

[0158] 1. Substituting the normal distribution prior and likelihood into the Bayesian formula, the posterior distribution is still a normal distribution (conjugate prior property): .

[0159] 1.1. Posterior Mean The expression is:

[0160]

[0161] in, (The measured value of memory resource usage corresponds to the mean of the actual running observation data); the numerator is the weighted sum of the prior mean and the observed mean, where the weight is the inverse of their respective variances; the denominator is the normalization coefficient.

[0162] 1.2. Posterior Variance The expression is:

[0163]

[0164] The posterior variance reflects the uncertainty after fusion: the smaller the prior variance and the measurement variance, the smaller the posterior variance (the more accurate the estimate).

[0165] Robust variance Function:

[0166] 1. Initial settings: (Memory resource usage has an uncertainty of 5% of the theoretical value);

[0167] 2. Dynamic adjustment (optional):

[0168] 2.1、If the measured data fluctuates greatly( Large), can be appropriately increased The expression for reducing the prior weight is:

[0169]

[0170] The purpose of changing the prior weight is to prevent the theoretical value from excessively dominating the posterior estimate due to memory resource usage.

[0171] The corresponding implementation steps of the Bayesian fusion algorithm are as follows:

[0172] A. Initialization:

[0173] A1. Input the theoretical value of memory resource usage , set the initial

[0174] A2. Initialize an empty observation list

[0175] The observation list is used to store the actual operation observation data of subsequent measured use cases.

[0176] B. Add actual operation observation data (actual value of memory resource usage):

[0177] B1. Actual measured value of memory resource usage each time Join the list

[0178] C. Calculate the posterior distribution:

[0179] C1. Calculate the mean of observations and the sample standard deviation

[0180] C2. Calculate according to the formula and

[0181] C3. Returns the posterior distribution parameters, i.e., the estimated value of memory resource usage (posterior mean + posterior variance)

[0182] D. Persistent metadata:

[0183] D1. Storage As the final memory resource usage estimate, as the confidence interval width.

[0184] For the implementation process of the above Bayesian fusion algorithm, a calculation example is provided:

[0185] Assumption: Theoretical value of memory resource usage M theory =1000KB, initial (KB)(5%), the sequence corresponding to the measured value of memory resource usage:

[0186] (KB).

[0187] The calculation steps are as follows:

[0188] 1. Calculation result of the average of the measured memory resource usage (estimated memory resource usage):

[0189] (KB).

[0190] 2. Calculation results of the variance of the measured values ​​of memory resource usage:

[0191] (KB 2 )

[0192] Among them, the standard deviation (KB)

[0193] 3. Calculation results of the posterior mean:

[0194]

[0195] Among them, the calculation result of the posterior variance is:

[0196]

[0197] Furthermore, the corresponding memory resource usage estimate is: $1002.1 \pm 8.16$ KB (95% confidence interval).

[0198] The core of the Bayesian fusion algorithm is to balance theoretical expectations with actual measurements by taking a weighted average of the theoretical and measured values ​​of memory resource usage. The robust variance described above is used to control the credibility of the theoretical values ​​of memory resource usage, preventing erroneous theoretical values ​​from dominating the estimation results of this application. Optionally, for dynamic adjustment, the prior weights can be adaptively adjusted based on the observed noise.

[0199] Compared to simply using theoretical memory usage values, this Bayesian fusion algorithm incorporates real-world observations (test cases) to effectively correct for deviations in theoretical memory usage values ​​caused by unmodeled factors such as JVM optimization and GC overhead (referring to the system resources consumed by the garbage collection process, such as CPU time, memory usage, and program response latency). This makes the memory usage estimates more consistent with real-world performance. Furthermore, compared to single or multiple estimates of memory usage, Bayesian fusion eliminates the impact of measurement noise and transient fluctuations through probabilistic modeling, providing estimates that are both realistic and statistically stable. The accompanying confidence intervals provide a clear indicator of prediction reliability, providing a quantitative basis for resource allocation decisions. This fusion mechanism overcomes the limitations of theoretical models that are often divorced from reality while avoiding the interference of random fluctuations in raw real-world observational data. In practical engineering, it achieves a balance between accuracy, robustness, and interpretability.

[0200] In the data analysis scenario of the financial field, in order to facilitate the management of the estimated memory resource usage, metadata is constructed based on the estimated memory resource usage. The metadata of the operator can then be constructed in the form of a directed acyclic graph (DAG). Optionally, the following provides a possible implementation method for persistent storage. Specifically, Figure 2 On the basis of Figure 5 A flow chart of another resource estimation method provided by an embodiment of the present invention is shown in FIG. Figure 5 , after step 104, further comprising:

[0201] Step 105: The estimated memory resource usage of each operator, the minimum amount of data required for each operator to run, and the basic operator information corresponding to each operator are combined to form the operator metadata of each operator.

[0202] The basic information of the operator may include but is not limited to: identification information of the operator and descriptive information of the operator, wherein the identification information of the operator is used to uniquely index the operator.

[0203] Step 106: Determine the operator corresponding to each data analysis directed acyclic graph (DAG) in all operator metadata.

[0204] Step 107: Determine the corresponding critical path according to the operator corresponding to each data analysis DAG.

[0205] The critical path represents the path with the largest memory resource usage in the data analysis DAG; each path is composed of multiple operators in the data analysis DAG. Specifically, Figure 6A schematic diagram of a data analysis DAG provided by an embodiment of the present invention is shown in FIG. Figure 6 , which consists of multiple vertices (operators) and directed edges. Operators include: Standard Deviation of Investment Return (I stands for Indicator, R stands for Return, and STD stands for Standard Deviation, abbreviated as IR_STD), Standard Deviation (STD), Indicator Return (IR), Profit and Loss (IPL), Profit and Loss (PL), Return Algorithm (DIV), and Cost (COST).

[0206] Among them, the following relationship exists between operators:

[0207] IR_STD and IR, STD: IR_STD depends on IR and STD, that is, the standard deviation of investment return rate is obtained by performing standard deviation operation on the investment return rate indicator, which can be expressed as IR_STD=STD(IR).

[0208] IR and IPL, DIV, COST: IR depends on IPL, DIV, and COST, that is, the investment return rate is obtained by dividing the profit or loss by the cost, IR=IPL / COST.

[0209] IPL and PL: IPL depends on PL, and the calculation of profit and loss indicators depends on the acquisition of profit and loss data.

[0210] Correspondingly, possible paths include:

[0211] Path 1 (IR_STD → IR → IPL → PL): This path involves multiple complex computing nodes, especially PL nodes, which may need to process a large amount of transaction data and may have high memory usage.

[0212] Path 2 (IR_STD → IR → DIV): This path involves IR_STD, IR, and DIV nodes and has a medium memory footprint.

[0213] Path 3 (IR_STD → IR → COST): This path involves the IR_STD, IR, and COST nodes and has a medium memory footprint.

[0214] Path 4 (IR_STD → STD): This path involves IR_STD STD nodes and has a high memory usage, especially the calculation of the STD nodes.

[0215] Step 108: Determine the estimated peak value of the data analysis DAG memory resource usage corresponding to the data analysis DAG based on all operators included in the critical path.

[0216] For example, see above Figure 6 Path 1 has the largest memory usage, making it the critical path in the data analysis DAG example. Therefore, based on all operators included in path 1: IR_STD, IR, IPL, and PL, we determine the estimated peak memory usage of the data analysis DAG.

[0217] It should be noted that steps 106 through 108 constitute a mechanism for dynamically planning the estimated peak memory usage of the data analysis DAG based on the data analysis DAG. This allows for dynamic estimation of the estimated peak memory usage of the corresponding data analysis DAG based on different data analysis DAGs, thereby further improving the matching of peak memory usage with memory resources.

[0218] Furthermore, when determining the target memory resource usage estimate, the DAG memory resource usage estimate peak value can be analyzed based on the data corresponding to the target computing request to determine the memory resource usage peak value corresponding to the target computing request. For details, see Figure 5 , step 109, comprising:

[0219] Step 109-1: When a target computing request is obtained, determine the peak memory resource usage corresponding to the target computing request based on the target execution parameter sequence of the target computing request, the data analysis DAG corresponding to the target computing request, and the estimated peak memory resource usage of the data analysis DAG.

[0220] Optionally, to facilitate logical management and monitoring of heap memory resources, the heap memory resources can be divided into an initial area (InitialHeap, abbreviated as: IH), a runtime area (RuntimeHeap, abbreviated as: RH), an available area (AvailableHeap, abbreviated as: AH), and a fault-tolerance area (Fault-toleranceHeap, abbreviated as: FH).

[0221] The initial area is the heap memory resources used after system startup; the running area is the estimated total resource usage of the target computing request; the available area is the heap memory resources available for system logic; and the fault tolerance area is the heap memory resources multiplied by the fault tolerance coefficient. The fault tolerance coefficient is a constant ratio used to tolerate fault estimation errors and GC fragmentation. Optionally, the initial value of the fault tolerance coefficient is set to 5%.

[0222] In one possible implementation, the above heap memory resources satisfy the following relationship:

[0223] Heap = IH + RH + AH + FH

[0224] The capacity of each partition is recorded and updated in real time in the memory, and the actual available heap memory is monitored through regular polling to correct the available zone.

[0225] Furthermore, the following provides a task scheduling mechanism based on the above heap memory resource division. Specifically, Figure 2 On the basis of Figure 7 A flow chart of another resource estimation method provided by an embodiment of the present invention is shown in FIG. Figure 7 , step 110, comprising:

[0226] Step 110 - 1 : Compare the available zone with the peak memory resource usage.

[0227] Specifically, if the available area is greater than the peak memory resource usage, step 111 is executed; if the available area is less than or equal to the peak memory resource usage, step 112 is executed.

[0228] Step 111: Call the computing engine to analyze and compute the target computing request.

[0229] Step 112: Put the target computing request into a buffer queue and wait.

[0230] Optionally, after step 111-1, the computing engine completes the analysis and calculations and updates the current request in the status cache as completed, updating AH = AH + M' and RH = RH – M' (the request is considered complete and all estimated resources for the request are reclaimable). M' is the peak memory resource usage in step 110-1. The buffer queue then dequeues and performs resource trial calculations in round-robin mode until AH > the next M' is satisfied. Step 110-1 and subsequent steps are repeated until the computing engine completes the analysis and calculations.

[0231] Alternatively, the computing engine is the component or framework responsible for executing computing tasks, used for computing process control, data processing, and algorithm execution. The computing engine is the primary location where OOMs occur in the system. While this patented solution minimizes OOMs, it still cannot guarantee 100% reliability. Therefore, a solution that automatically recovers and adjusts the prediction model is required.

[0232] Optionally, the following provides an implementation scheme for automatically recovering and adjusting the estimation model:

[0233] First, calculate the configuration parameters of the computing engine.

[0234] For example, XX:OnOutOfMemoryError="restart_script.sh". This configuration parameter is used to automatically restart the system when an OOM error occurs. "restart_script.sh" is the system restart script. When the script is executed, the OOM restart information is recorded.

[0235] Furthermore, after the OOM system restarts, the above-mentioned fault tolerance coefficient is increased with an adjustment step of 5% and an upper limit of 20%. If OOM still occurs after reaching the upper limit, the system rejects similar requests and sends a notification to the operation and maintenance management for intervention.

[0236] Finally, after successfully adjusting the fault tolerance coefficient, the above robust variance is increased, the step size is set to 1%, and the posterior is automatically updated iteratively.

[0237] In order to achieve the various steps and corresponding technical effects in the above example, a possible implementation method of a resource estimation device is provided below. Specifically, Figure 8 A schematic diagram of a resource estimation device provided by an embodiment of the present invention, see Figure 8 The resource estimation device 200 includes: a resource estimation model 201 and a task manager 202.

[0238] Resource estimation model 201 is used to estimate the theoretical value of memory resource usage of any operator during the analysis and calculation process; an operator corresponds to the minimum computing unit for data in the analysis and calculation process; based on the measured use cases and the corresponding execution parameter sequence, the measured value of memory resource usage of the corresponding operator during the analysis and calculation process is estimated; Bayesian fusion is performed on the theoretical value of memory resource usage and the corresponding measured value of memory resource usage to obtain the estimated value of memory resource usage of the operator;

[0239] The task manager 202 is used to determine the peak memory resource occupancy corresponding to the target computing request based on the target execution parameter sequence of the target computing request, at least one target operator corresponding to the target computing request, and the estimated target memory resource occupancy corresponding to all target operators when obtaining the target computing request; determine whether the current memory resource capacity meets the peak memory resource occupancy, and if so, call the computing engine to analyze and calculate the target computing request; if not, put the target computing request into the buffer queue for waiting.

[0240] Optionally, the resource estimation model 201 is specifically used to:

[0241] Recursively traverse the abstract syntax tree corresponding to each operator to obtain the recursive level information corresponding to each operator; obtain the theoretical resource occupancy value and the number of objects corresponding to each operator; determine the theoretical memory resource occupancy value based on the recursive level information, the theoretical resource occupancy value of each object, and the number of objects.

[0242] Optionally, the resource estimation model 201 is specifically used to:

[0243] The data scale of the measured use case is determined by setting the data scale steps of the measured use case and the corresponding execution parameter sequence; the data scale is input into the linear regression model to obtain the corresponding measured value of memory resource usage.

[0244] Optionally, the resource estimation model 201 is specifically used to:

[0245] Perform a Bayesian fusion of the theoretical memory resource usage values ​​and the corresponding measured memory resource usage values ​​to obtain the posterior distribution. Take the mean of the posterior distribution to obtain the estimated memory resource usage of the operator.

[0246] In order to facilitate the management of the estimated value of memory resource occupancy, in a possible implementation, metadata is formed based on the estimated value of memory resource occupancy. Specifically, Figure 8 On the basis of Figure 9 A schematic diagram of another resource estimation device provided by an embodiment of the present invention, see Figure 9 The resource estimation device 200 also includes: a metadata manager 203 and a resource manager 204.

[0247] The metadata manager 203 is used to form the operator metadata of each operator by combining the estimated memory resource occupancy of each operator, the minimum amount of data required for each operator to run, and the basic operator information corresponding to each operator; among all the operator metadata, the operator corresponding to each data analysis directed acyclic graph (DAG) is determined; the corresponding critical path is determined according to the operator corresponding to each data analysis DAG; the critical path represents the path with the maximum memory resource occupancy in the corresponding data analysis DAG; each path is composed of multiple operators in the data analysis DAG; based on all the operators included in the critical path, the estimated peak value of the memory resource occupancy of the data analysis DAG corresponding to the data analysis DAG is determined.

[0248] Optionally, the task manager 202 is specifically configured to:

[0249] When a target computing request is obtained, the peak memory resource usage corresponding to the target computing request is determined based on the target execution parameter sequence of the target computing request, the data analysis DAG corresponding to the target computing request, and the estimated peak memory resource usage of the data analysis DAG.

[0250] Optionally, the resource manager 204 is used to divide the heap memory resources into an initial area (IH), a running area (RH), an available area (AH) and a fault tolerance area (FH); wherein the initial area is the heap memory resources used after the system is started; the running area is the estimated total value of the resource occupancy requested by the target calculation; the available area is the heap memory resources available for system logic; and the fault tolerance area is the heap memory resources multiplied by the fault tolerance coefficient, where the fault tolerance coefficient is a constant ratio.

[0251] Optionally, the resource manager 204 is specifically configured to:

[0252] Compare the available zone with the peak memory resource usage; if the available zone is greater than the peak memory resource usage, call the computing engine to analyze and calculate the target computing request; if the available zone is less than or equal to the peak memory resource usage, put the target computing request into the buffer queue for waiting.

[0253] In order to more fully illustrate the resource estimation device shown in the above embodiment of the present application, the specific functions and data flows of each model and module of the resource estimation device are exemplified below. Figure 9 On the basis of Figure 10 A schematic diagram of another resource estimation device provided by an embodiment of the present invention, see Figure 10 The resource estimation device 200 also includes: a database 205 and a calculation engine 206.

[0254] Specifically, the resource estimation model 201 can use the "theoretical calculation method" described above to obtain the theoretical value of memory resource usage for any operator during the analysis and calculation process. Furthermore, the "measured regression method" described above can be used to obtain the measured value of memory resource usage for the corresponding operator during the analysis and calculation process. Furthermore, through "Bayesian fusion," the theoretical value of memory resource usage and the measured value of memory resource usage are fused to form an estimated value of memory resource usage for the operator.

[0255] The metadata manager 203 forms the operator metadata of each operator by combining the estimated memory resource usage of each operator, the minimum amount of data required for each operator to run, and basic operator information corresponding to each operator.

[0256] Furthermore, since the data analysis DAG may cover multiple operators, the metadata manager 203 may convert the operator metadata into DAG metadata corresponding to the data analysis DAG based on the data format of the data analysis DAG.

[0257] Optionally, the metadata manager 203 may persistently store the operator metadata and the DAG metadata as metadata in the database 205 for subsequent reuse.

[0258] Optionally, the database 205 may also store actual operation observation data, ie, the measured data shown in the figure, and metadata corresponding to all objects and their attribute structure information.

[0259] Optionally, during the "dynamic planning" shown above, the resource estimation model 201 can call the corresponding operator metadata from the metadata manager 203 to determine the operator corresponding to each data analysis directed acyclic graph DAG, and then determine the corresponding critical path based on the operator corresponding to each data analysis DAG; based on all the operators contained in the critical path, determine the estimated peak value of the data analysis DAG memory resource occupancy corresponding to the data analysis DAG.

[0260] Optionally, in the “initialization” mentioned in the above example, the metadata manager 203 may be initialized through the database 205 .

[0261] Specifically, the initialization may refer to: reading metadata from the database 205 and storing it in the program memory as a cache, thereby increasing access speed and reducing direct access to the database 205 .

[0262] Optionally, upon receiving a target computing request, the task manager 202 may perform a resource trial calculation, i.e., step 109 described above. Optionally, the target execution parameter sequence, at least one target operator corresponding to the target computing request, and the target memory resource usage estimates corresponding to all target operators involved in step 109 may all be obtained from the metadata manager 203. This may be operator metadata obtained at the operator granularity or DAG metadata obtained at the DAG granularity.

[0263] Then, the task manager 202 determines whether resources are sufficient through step 110 above. If resources are insufficient, the target computing request is placed in a buffer queue. If resources are sufficient, the computing engine 206 is called to analyze and compute the target computing request. After the computation is complete, the computing engine 206 updates the status cache.

[0264] The embodiment of the present invention further provides an electronic device that can execute all the steps of the above examples of the embodiment of the present invention to achieve the corresponding technical effects. Specifically, Figure 11 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention, see Figure 11 , the electronic device 30 , comprises: a memory 301 , a processor 300 ;

[0265] Memory 301, used to store one or more programs;

[0266] Processor 3300;

[0267] When one or more programs are executed by the processor, when the electronic device 30 is used to perform the steps shown in the above-mentioned method examples, it can achieve each step and corresponding technical effects.

[0268] In the embodiments provided in this application, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architectures, functions and operations of the devices, methods and computer program products according to multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of the code, and the module, program segment or a part of the code contains one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or action, or can be implemented using a combination of dedicated hardware and computer instructions.

[0269] In addition, the functional modules in each embodiment of the present application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0270] If the function is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the existing technology, or the part of the technical solution, can be embodied in the form of a program product. The program product is stored in a computer-readable storage medium and includes a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, and other media that can store program code.

[0271] Compared to existing technologies that use Spark or Flink as heavyweight computing engines, the solution provided by the above examples of the present invention can more lightweight, flexible, and accurately estimate the resources corresponding to computing tasks, thereby quantifying the scheduling of computing tasks. This solves the problem that existing solutions cannot estimate the resource requirements of computing requests, or the estimated resources deviate significantly from the actual resources and resource requirements.

[0272] At the same time, even if OOM occurs, without the need for manual intervention and tuning and the introduction of heavyweight middleware, the above-mentioned fault tolerance coefficient and robust variance can be increased to iterate the estimation mechanism of this solution, further reducing the OOM risk and the overall cost of using the solution.

[0273] During stress load testing and capacity testing, by comparing JVM memory usage data with estimated data, the root mean square error (RMSE) was around 3%-7%, a relatively low error level. In actual production applications, this solution effectively ensures system reliability and reduces the probability of OOM.

[0274] In this patented solution, the direction and magnitude of Bayesian fusion results are adjusted based on their relative credibility: if the theoretical value variance (robust variance) is small, the fusion result is closer to the theoretical value; if the measured value variance is small (accurate measurement), the measured value is favored. This adaptive weighting makes Bayesian fusion suitable for quantifying uncertainty in memory resource usage, and can be used to design a wider range of memory resource management quantification solutions.

[0275] The above are merely preferred embodiments of the present application and are not intended to limit the present application. Those skilled in the art will readily appreciate that various modifications and variations are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present application shall be included within the scope of protection of the present application.

[0276] It will be apparent to those skilled in the art that the present application is not limited to the details of the exemplary embodiments described above and that the present application can be implemented in other specific forms without departing from the spirit or essential characteristics of the present application. Therefore, the embodiments should be considered in all respects as illustrative and non-restrictive, and the scope of the present application is defined by the appended claims, not the foregoing description, and all variations within the meaning and range of equivalents of the claims are intended to be included therein. Any reference sign in a claim should not be construed as limiting the claim to which it relates.

Claims

1. A resource estimation method, characterized in that: include: Estimate the theoretical value of memory resource usage of any operator during the analysis and calculation process; The operator corresponds to the smallest calculation unit for data in the analysis and calculation process; Estimate the measured value of memory resource usage of the corresponding operator during the analysis and calculation process based on the measured use case and the corresponding execution parameter sequence; the measured use case includes one or more batches of actual operation observation data; The execution parameter sequence is used to reflect the parameter information corresponding to the measured use case; Performing Bayesian fusion on the theoretical value of memory resource occupancy and the corresponding measured value of memory resource occupancy to obtain an estimated value of memory resource occupancy of the operator; When a target computing request is obtained, determining a peak memory resource occupancy value corresponding to the target computing request based on a target execution parameter sequence of the target computing request, at least one target operator corresponding to the target computing request, and target memory resource occupancy estimates corresponding to all the target operators; Determine whether the current memory resource capacity meets the memory resource occupancy peak, and if so, call the computing engine to analyze and calculate the target computing request; If not, the target computing request is put into a buffer queue and waits.

2. The method according to claim 1, characterized in that The step of estimating the theoretical value of memory resource usage of any operator during the analysis and calculation process includes: Recursively traverse the abstract syntax tree corresponding to each operator to obtain the recursive level information corresponding to each operator; Obtaining theoretical resource occupancy values ​​and object quantities of various objects corresponding to each operator; The theoretical value of memory resource occupation is determined according to the recursive level information, the theoretical value of resource occupation of each type of object, and the number of objects.

3. The method according to claim 1, characterized in that The step of estimating the measured value of memory resource usage of the corresponding operator during the analysis and calculation process based on the measured use case and the corresponding execution parameter sequence includes: By testing the data scale ladder and the corresponding execution parameter sequence of the measured use case, the measured peak value of the memory resource usage of the measured use case is obtained; The data scale is used as the independent variable, and the measured peak value of the memory resource occupancy is used as the dependent variable, and the corresponding measured value of the memory resource occupancy is obtained through a linear regression model.

4. The method according to claim 1, wherein The step of performing Bayesian fusion on the theoretical value of memory resource occupancy and the corresponding measured value of memory resource occupancy to obtain an estimated value of memory resource occupancy of the operator includes: Performing Bayesian fusion on the theoretical value of memory resource occupancy and the corresponding measured value of memory resource occupancy to obtain a posterior distribution; The posterior distribution is averaged to obtain an estimated memory resource occupancy value of the operator.

5. The method according to claim 1, wherein After the step of performing Bayesian fusion on the theoretical value of memory resource occupancy and the corresponding measured value of memory resource occupancy to obtain an estimated value of memory resource occupancy of the operator, the method further includes: The estimated memory resource usage of each operator, the minimum amount of data required for each operator to run, and the basic operator information corresponding to each operator are used to form the operator metadata of each operator; Determine the operator corresponding to each data analysis DAG in all the operator metadata; Determine a corresponding critical path based on the operator corresponding to each data analysis DAG; the critical path represents the path with the largest memory resource usage in the data analysis DAG; each path is composed of multiple operators in the data analysis DAG; Determine an estimated peak value of data analysis DAG memory resource usage corresponding to the data analysis DAG based on all operators included in the critical path.

6. The method according to claim 5, characterized in that When a target computing request is obtained, the step of determining a peak memory resource occupancy corresponding to the target computing request based on a target execution parameter sequence of the target computing request, at least one target operator corresponding to the target computing request, and target memory resource occupancy estimates corresponding to all the target operators includes: When a target computing request is obtained, the peak memory resource occupancy corresponding to the target computing request is determined based on the target execution parameter sequence of the target computing request, the data analysis DAG corresponding to the target computing request, and the estimated peak memory resource occupancy of the data analysis DAG.

7. The method according to claim 1, characterized in that Also includes: Divide the heap memory resources into an initial area, a running area, an available area, and a fault-tolerant area; wherein the initial area is the heap memory resources used after the system is started; The running area is the estimated total value of resources occupied by the target computing request; the available area is the heap memory resources available for system logic; the fault tolerance area is the heap memory resources multiplied by the fault tolerance coefficient, and the fault tolerance coefficient is a constant ratio.

8. The method according to claim 7, characterized in that The step of determining whether the current memory resource capacity meets the peak memory resource occupancy includes: Comparing the available zone with the peak memory resource usage; If the available area is larger than the peak memory resource usage, calling the computing engine to analyze and calculate the target computing request; If the available area is less than or equal to the peak memory resource occupancy, the target computing request is placed in a buffer queue for waiting.

9. A resource estimation device, characterized in that: include: Resource estimation model and task manager; The resource estimation model is used to estimate the theoretical value of memory resource usage of any operator during the analysis and calculation process; The operator corresponds to the minimum computing unit for data in the analysis and calculation process; based on the measured use case and the corresponding execution parameter sequence, the measured value of memory resource usage of the corresponding operator in the analysis and calculation process is estimated; Performing Bayesian fusion on the theoretical value of memory resource occupancy and the corresponding measured value of memory resource occupancy to obtain an estimated value of memory resource occupancy of the operator; the measured use case includes one or more batches of actual operation observation data; The execution parameter sequence is used to reflect the parameter information corresponding to the measured use case; The task manager is configured to, when receiving a target computing request, determine a peak memory resource occupancy value corresponding to the target computing request based on a target execution parameter sequence of the target computing request, at least one target operator corresponding to the target computing request, and target memory resource occupancy estimates corresponding to all the target operators; Determine whether the current memory resource capacity meets the memory resource occupancy peak. If so, call the computing engine to analyze and calculate the target computing request; if not, put the target computing request into a buffer queue for waiting.

10. The device according to claim 9, characterized in that Also includes: Metadata Manager; The metadata manager is configured to form operator metadata for each operator by combining an estimated memory resource usage of each operator, a minimum amount of data required for each operator to run, and basic operator information corresponding to each operator; Determine the operator corresponding to each data analysis DAG in all the operator metadata; Determine the corresponding critical path based on the operator corresponding to each data analysis DAG; the critical path represents the path with the maximum memory resource occupancy in the data analysis DAG; each path is composed of multiple operators in the data analysis DAG; based on all operators included in the critical path, determine the estimated peak value of the data analysis DAG memory resource occupancy corresponding to the data analysis DAG.

11. The device according to claim 9, characterized in that Also includes: Explorer; The resource manager is used to divide heap memory resources into an initial area, a running area, an available area, and a fault-tolerant area; wherein the initial area is the heap memory resources used after the system is started; the running area is the estimated total resource occupancy of the target computing request; the available area is the heap memory resources available for system logic; and the fault-tolerant area is the heap memory resources multiplied by a fault-tolerant coefficient, where the fault-tolerant coefficient is a constant ratio.

12. An electronic device, characterized in that: include: a memory for storing one or more programs; processor; When the one or more programs are executed by the processor, the method according to any one of claims 1 to 8 is implemented.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

14. A program product, characterized in that When the program product is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images